Thermal control of data center

ABSTRACT

A method includes measuring a plurality of temperatures corresponding to a plurality of servers located in a data center, determining a subset of the plurality of servers as high-temperature servers based on the plurality of temperatures, and reassigning tasks from at least a portion of the subset of the plurality of servers to one or more other servers of the plurality of servers.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.16/579,686 filed Sep. 23, 2019, which claims the benefit of and priorityto U.S. Patent Application No. 62/734,829 filed Sep. 21, 2018, theentire disclosures of which are incorporated by reference herein.

BACKGROUND

The present disclosure relates to HVAC control for a data center, andmore particularly to HVAC operation based on thermal contributions ofservers and other data center equipment. Data center equipment (e.g.,servers, processors, routers, power supplies, etc.) generate heat due toelectrical resistance of electronic communications transmitted andprocessed therein. Building equipment, such as HVAC systems and otherbuilding systems, operate to manage the temperature and/or otherenvironmental conditions in a space, for example a data center thathouses data center equipment. Accordingly, a need exists for HVACcontrol systems that account for heat generated by data centerequipment.

Some embodiments of the present disclosure relate generally to theoperation of a central plant for serving building thermal energy loadsand to distributing building thermal energy loads across a plurality ofsubplants configured to serve the building thermal energy loads. Itshould be understood that concepts, features, functions, processes,controllers, circuits, etc. are described herein with reference to acentral plant and/or the plurality of subplants for the sake ofgenerality and breadth and may also be adapted to be applied directly toan HVAC system, for example an HVAC system that serves a data center.The term HVAC system or equipment refers to a system associated withheating, ventilation, and/or air conditioning operations.

A central plant may include various types of equipment configured toserve the thermal energy loads of a building or campus (i.e., a systemof buildings). For example, a central plant may include heaters,chillers, heat recovery chillers, cooling towers, or other types ofequipment configured to provide heating or cooling for the building. Acentral plant may consume resources from a utility (e.g., electricity,water, natural gas, etc.) to heat or cool a working fluid (e.g., water,glycol, etc.) that is circulated to the building or stored for later useto provide heating or cooling for the building. Fluid conduits typicallydeliver the heated or chilled fluid to air handlers located on therooftop of the building or to individual floors or zones of thebuilding. The air handlers push air past heat exchangers (e.g., heatingcoils or cooling coils) through which the working fluid flows to provideheating or cooling to the air. The working fluid then returns to thecentral plant to receive further heating or cooling and the cyclecontinues.

High efficiency equipment can help reduce the amount of energy consumedby a central plant; however, the effectiveness of such equipment ishighly dependent on the control technology that is used to distributethe load across the multiple subplants. For example, it may be more costefficient to run heat pump chillers instead of conventional chillers anda water heater when energy prices are high. It is difficult andchallenging to determine when and to what extent each of the multiplesubplants should be used to minimize energy cost. If electrical demandcharges are considered, the optimization is even more complicated.

Thermal energy storage can be used to store energy for later use. Whencoupled with real-time pricing for electricity and demand charges,thermal energy storage provides a degree of flexibility that can be usedto greatly decrease energy costs by shifting production to low costtimes or when other electrical loads are lower so that a new peak demandis not set. It is difficult and challenging to integrate thermal energystorage with a central plant having multiple subplants and to optimizethe use of thermal energy storage in conjunction with the multiplesubplants to minimize energy cost.

SUMMARY

One implementation of the present disclosure is a method for operatingheating, ventilation and air conditioning (HVAC) system in a datacenter. The method includes removing heat from air in the data centerutilizing the HVAC system; collecting space temperature data and servertemperature data, and using the space temperature data, the servertemperature data, and a model of performance for the HVAC system and thedata center to predict changes to the HVAC system and the data center.The changes are predicted to conserve energy while complying withtemperature constraints for the data center and meeting a processingdemand on the data center. The method also includes electronicallycontrolling the HVAC system and the data center in accordance with thepredicted changes for both the HVAC system and the data center.

In some embodiments, the model of performance of the HVAC system and thedata center models an operational efficiency of servers of the datacenter as a function of at least one of the space temperature data orthe server temperature data. In some embodiments, the model ofperformance for the HVAC system and the data center models thermalbehavior of the data center as a function of a predicted outdoor airtemperature.

In some embodiments, the changes include operating the HVAC system topre-cool the data center in advance of a predicted increase in theprocessing demand on the data center. In some embodiments, the changesinclude shifting one or more computing tasks of the data center to atime period corresponding to a predicted increase in the efficiency ofthe HVAC system.

In some embodiments, the temperature constraints for the data centerinclude a plurality of server temperature constraints corresponding to aplurality of servers. In some embodiments, the changes include providingtargeted cooling to a first server of the data center and reassigningtasks from a second server of the data center to a third server of thedata center.

Another implementation of the present disclosure is a method. The methodincludes measuring a plurality of temperatures corresponding to aplurality of servers located in a data center, determining a subset ofthe plurality of servers as high-temperature servers based on theplurality of temperatures, and reassigning tasks from at least a portionof the subset of the plurality of servers to one or more other serversof the plurality of servers.

In some embodiments, determining a subset of the plurality of servers ashigh-temperature servers based on the plurality of temperatures includescomparing, for each of the plurality of servers, a setpoint for theserver to a corresponding temperature of the plurality of temperatures.

In some embodiments, the method includes providing targeted cooling toone or more servers in the subset. In some embodiments, reassigningtasks from at least a portion of the subset of the plurality of serversto one or more other servers of the plurality of servers comprisescontrolling the plurality of temperatures towards equilibrium across thedata center.

In some embodiments, the method includes predicting a high-activityperiod for the plurality of servers and pre-cooling the data center inadvance of the high-activity period. In some embodiments, the methodincludes rescheduling the tasks based on a time-variant cost ofoperating building equipment to cool the data center. In someembodiments, the one or more other servers have temperatures at or belowa preferred operating temperature.

Another implementation of the present disclosure is a system. The systemincludes servers located at a data center and equipment configured tocool the servers. The equipment includes at least one of a centralplant, an airside system, a waterside system, rack cooling equipment, acomputer-room air conditioner, a rooftop unit, a floor cooling system,or a liquid cooling system. The system also includes a controllerconfigured to control the equipment based on estimates of amounts ofthermal energy output by each of the servers.

In some embodiments, the controller is configured to apply the estimatesof the amounts of thermal energy output by the servers in a feedforwardcontrol approach to generate setpoints for the equipment.

In some embodiments, estimates of the amounts thermal energy output bythe servers comprise predictions of the amount of thermal energy outputby the servers for a plurality of time steps. The controller isconfigured to generate setpoints for the equipment for the plurality oftime steps based on the predictions. In some embodiments, the controlleris configured to generate the setpoints by performing an optimizationprocess that minimizes an overall resource usage of the servers and theequipment over a time horizon.

In some embodiments, the controller is configured to rescheduleoperations of the servers based on a time-variant cost of operating theequipment. In some embodiments, the servers include fans and thecontroller is configured to control the fans. In some embodiments, thecontroller is configured to coordinate control of the waterside system,the airside system, the rack-cooling system, and the fans.

Another implementation of the present disclosure is a method. The methodincludes controlling operations of servers at a data center, controllingbuilding equipment to cool the servers, determining a high-cost periodfor operating the building equipment and a low-cost period for operatingthe building equipment, identifying a time-insensitive task scheduled tooccur during the high-cost period, and causing the time-insensitive taskto be executed during the low-cost period.

In some embodiments, the method includes predicting a high-activityperiod for the servers and operating the building equipment to reduce atemperature at the servers before the high-activity period.

In some embodiments, the method includes comprising shifting thetime-insensitive task from a first server to a second server based on atemperature differential between the first server and the second server.In some embodiments, determining the high-cost period includespredicting weather-based loads on the building equipment. In someembodiments, determining the high-cost period and the low-cost periodincludes determining utility rates over a time horizon. In someembodiments, the building equipment includes at least one of a centralplant, an airside system, a waterside system, rack cooling equipment, acomputer-room air conditioner, a rooftop unit, a floor cooling system,or a liquid cooling system.

Another implementation of the present disclosure is a method. The methodincludes providing a model of thermal behavior of a data center. Themodel includes a data center equipment term representing an amount ofthermal energy provided to the data center by data center equipment. Themethod includes predicting values of the data center equipment term overa time horizon and performing an optimization process using the valuesand the model to generate setpoints for building equipment over the timehorizon. The method also includes controlling the building equipment inaccordance with the setpoints.

In some embodiments, the setpoints include amounts of thermal energy tobe provided to the data center by the building equipment. In someembodiments, controlling the building equipment in accordance with thesetpoints comprises pre-cooling the data center in advance of apredicted high-activity period for the data center equipment.

In some embodiments, the method includes coordinating control of thedata center equipment and the building equipment. Coordinating controlof the data center equipment and the building equipment may includescheduling operation of the data center equipment to reduce the amountof thermal energy provided to the data center by the building equipmentover the time horizon. Coordinating control of the data center equipmentand the building equipment may include scheduling operation of the datacenter based on a time-variant cost of operating the building equipment.Coordinating control of the data center equipment and the buildingequipment may include controlling the data center equipment to generateheat to reduce a load on the building equipment.

In some embodiments, predicting values of the data center equipment termcomprises predicting heat generated by the data center equipment as afunction of complexity of calculations executed on the data centerequipment. In some embodiments, predicting values of the data centerequipment term comprises predicting heat generated by the data centerequipment based on a set of upcoming operations to be performed by thedata equipment. In some embodiments, predicting values of the datacenter equipment term comprises predicting heat generated by the datacenter equipment as a function of physical data transmission routes ofpredicted data transmissions in the data center equipment.

In some embodiments, the data center equipment term is implemented inthe model of thermal behavior of the data center as an input disturbancemodel. In some embodiments, providing the model comprises performing asystem identification process using training data, the training datarepresenting historical behavior of the building equipment and the datacenter equipment.

Another implementation of the present disclosure is one or morenon-transitory computer-readable media containing program instructionsthat, when executed by one or more processors, cause the one or moreprocessors to perform operations. The operations include predictingamounts of thermal energy provided by a data center equipment atspecific times, receiving a measured temperature associated with thedata center, using a model of thermal behavior of the data center toheat or cool the data center using building equipment. The model usesthe measured temperature and the amounts of thermal energy provided bythe data center to control the building equipment.

In some embodiments, the model operates to pre-cool the data center inadvance of a predicted high-activity period for the data center. In someembodiments, the operations further comprising coordinating control ofthe data center equipment and the building equipment. Coordinatingcontrol of the data center equipment and the building equipment mayinclude scheduling operation of the data center to reduce the amount ofthermal energy provided to the data center by the building equipmentover the time horizon. Coordinating control of the data center equipmentand the building equipment may include scheduling operation of the datacenter based on a time-variant cost of operating the building equipment.

In some embodiments, predicting values of the data center equipment termcomprises predicting heat generated by the data center equipment as afunction of complexity of calculations executed on the data centerequipment. In some embodiments, predicting amounts of thermal energyprovided by data center equipment comprises predicting heat generated bythe data center equipment based on a set of upcoming operations to beperformed by the data equipment. In some embodiments, predicting amountsof thermal energy provided by data center equipment comprises predictingheat generated by the data center equipment as a function of physicaldata transmission routes of predicted data transmissions in the datacenter equipment.

Another implementation of the present disclosure is a heating,ventilation, and cooling (HVAC) system. The HVAC system includesbuilding equipment and a controller. The controller is configured toreceive a temperature value, a building mass heat value, and aprediction of heat provided by data center equipment in a data centerand use a model to control the building equipment to heat or cool thedata center. The model uses the building mass heat value and theprediction of heat provided by the thermal equipment to control thebuilding equipment.

In some embodiments, the controller is configured to use the model toschedule amounts of heat to be provided or removed from the data centerby the building system at a plurality of time steps. In someembodiments, the controller is configured to perform a systemidentification process to identify the model using training data. Thetraining data may include historical information relating to operationof the data center equipment. The training data may include historicalinformation relating to operating of the building equipment andhistorical weather data.

In some embodiments, the controller is configured to use the model tocontrol the building equipment by generating setpoints for the buildingequipment that minimize a cost predicted using the model.

In some embodiments, the controller is further configured to coordinatecontrol of the data center equipment and the building equipment using anartificial intelligence agent. In some embodiments, the controller isconfigured to schedule operation of the data center equipment to reducethe amount of thermal energy removed from the data center by thebuilding equipment over a time period. In some embodiments, thecontroller is configured to schedule operation of the data centerequipment based on a time-variant cost of operating the buildingequipment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a central plant having a plurality ofsubplants including a heater subplant, heat recovery chiller subplant, achiller subplant, a hot thermal energy storage subplant, and a coldthermal energy storage subplant, according to an exemplary embodiment.

FIG. 2 is a block diagram illustrating a central plant system includinga central plant controller that may be used to control the central plantof FIG. 1 , according to an exemplary embodiment.

FIG. 3 is block diagram illustrating a portion of central plant systemof FIG. 2 in greater detail, showing a load/rate prediction circuit, ahigh level optimization circuit, a low level optimization circuit, abuilding automation system, and central plant equipment, according to anexemplary embodiment.

FIG. 4 , a block diagram illustrating the high level optimizationcircuit of FIG. 3 in greater detail, according to an exemplaryembodiment.

FIGS. 5A-5B are subplant curves illustrating a relationship between theresource consumption of a subplant and the subplant load and which maybe used by the high level optimization circuit of FIG. 4 to optimize theperformance of the central plant of FIG. 1 , according to an exemplaryembodiment.

FIG. 6 is a non-convex and nonlinear subplant curve that may begenerated from experimental data or by combining equipment curves forindividual devices of the central plant, according to an exemplaryembodiment.

FIG. 7 is a linearized subplant curve that may be generated from thesubplant curve of FIG. 6 by converting the non-convex and nonlinearsubplant curve into piecewise linear segments, according to an exemplaryembodiment.

FIG. 8 is a graph illustrating a set of subplant curves that may begenerated by the high level optimization circuit of FIG. 3 based onexperimental data from a low level optimization circuit for multipledifferent environmental conditions, according to an exemplaryembodiment.

FIG. 9 is a block diagram of a planning system that incorporates thehigh level optimization circuit of FIG. 3 , according to an exemplaryembodiment.

FIG. 10 is a drawing illustrating the operation of the planning systemof FIG. 9 , according to an exemplary embodiment.

FIG. 11 is a flowchart of a process for optimizing cost in a centralplant that may be performed by the central plant controller of FIG. 2 orthe planning system of FIG. 9 , according to an exemplary embodiment.

FIG. 12 is a block diagram of a data center, according to an exemplaryembodiment.

FIG. 13 is a circuit-style diagram of heat transfer in the data centerof FIG. 12 , according to an exemplary embodiment.

FIG. 14 is a block diagram of a controller of a HVAC system for use withdata center of FIG. 12 , according to an exemplary embodiment.

FIG. 15 is a block diagram of an integrated controller for use with aHVAC system and data center equipment of the data center of FIG. 12 ,according to an exemplary embodiment.

FIG. 16 is a block diagram of a system for feedforward control,according to an exemplary embodiment.

FIG. 17 is a flowchart of a process for controlling thermal behavior ofthe data center, according to an exemplary embodiment.

FIG. 18 is a flowchart of a process for pre-cooling a data center,according to an exemplary embodiment.

FIG. 19 is a flowchart of a process for scheduling operations of thedata center equipment, according to an exemplary embodiment.

FIG. 20 is a flowchart of a process for controlling thermal behavior ofthe data center, according to an exemplary embodiment.

FIG. 21 is a block diagram of a system of controlling thermal behaviorof the data center, according to an exemplary embodiment.

FIG. 22 is schematic illustration of a liquid cooling system for a datacenter, according to an exemplary embodiment.

FIG. 23 is an illustration of a heat map for a data center andassociated approaches for managing the heat equilibrium, according to anexemplary embodiment.

DETAILED DESCRIPTION

Overview

Referring generally to the FIGURES, systems and methods for controllingHVAC equipment to heat or cool a data room (data center) are shown,according to exemplary embodiments. In some embodiments, systems andmethods for optimizing control of a central plant and/or an HVAC systemare shown, according to an exemplary embodiment. The systems and methodsof the present disclosure are particularly suited for optimizing acentral plant and/or an HVAC system that serves a data center, i.e., aspace of a building that stores data center equipment (e.g., servers,computers, processors, routers, network components, etc.), in someembodiments. Data center equipment generates heat via electricalresistance while operating to execute various computing functions. Thisheat contributes to a disturbance load on the data center, therebyaffecting the behavior of the central plant and/or HVAC system. Thesystems and methods described herein allow for coordination between acentral plant and/or HVAC system and the data center equipment toimprove optimization of the operation of both the central plant/HVACsystem and the data center equipment, in some embodiments. For example,predictions of the heat generated by the data center equipment may beused in high level optimization, low level optimization, and/or modelpredictive control of a central plant and/or HVAC system. In someembodiments, a data center HVAC system which is not a central plantoperates to heat or cool the data center. For example, a dedicated HVACunit or series of units could provide heating and cooling for the datacenter.

A central plant is one type of system that could be used to heat or coola data center. A central plant may include may include various types ofequipment configured to serve the thermal energy loads of a building orcampus (i.e., a system of buildings). For example, a central plant mayinclude heaters, chillers, heat recovery chillers, cooling towers, orother types of equipment configured to provide heating or cooling forthe building or campus. The central plant equipment may be divided intovarious groups configured to perform a particular function. Such groupsof central plant equipment are referred to herein as subplants. Forexample, a central plant may include a heater subplant, a chillersubplant, a heat recovery chiller subplant, a cold thermal energystorage subplant, a hot thermal energy storage subplant, etc. Thesubplants may consume resources from one or more utilities (e.g., water,electricity, natural gas, etc.) to serve the energy loads of thebuilding or campus. Optimizing the central plant may include operatingthe various subplants in such a way that results in a minimum monetarycost to serve the building energy loads.

In some embodiments, the central plant optimization is a cascadedoptimization process including a high level optimization and a low leveloptimization. The high level optimization may determine an optimaldistribution of energy loads across the various subplants. For example,the high level optimization may determine a thermal energy load to beproduced by each of the subplants at each time element in anoptimization period. In some embodiments, the high level optimizationincludes optimizing a high level cost function that expresses themonetary cost of operating the subplants as a function of the resourcesconsumed by the subplants at each time element of the optimizationperiod. The low level optimization may use the optimal load distributiondetermined by the high level optimization to determine optimal operatingstatuses for individual devices within each subplant. Optimal operatingstatuses may include, for example, on/off states and/or operatingsetpoints for individual devices of each subplant. The low leveloptimization may include optimizing a low level cost function thatexpresses the energy consumption of a subplant as a function of theon/off states and/or operating setpoints for the individual devices ofthe subplant.

In some embodiments, high level optimization systems and methods areprovided for controlling HVAC equipment. A high level optimizationcircuit may perform the high level optimization. In various embodiments,the high level optimization circuit may be a component of a centralplant controller configured for real-time control of a physical plant ora component of a planning tool configured to optimize a simulated plant(e.g., for planning or design purposes).

In some embodiments, the high level optimization circuit uses a linearprogramming framework to perform the high level optimization.Advantageously, linear programming can efficiently handle complexoptimization scenarios and can optimize over a relatively longoptimization period (e.g., days, weeks, years, etc.) in a relativelyshort timeframe (e.g., seconds, milliseconds, etc.). In otherembodiments, the high level optimization circuit may use any of avariety of other optimization frameworks (e.g., quadratic programming,linear-fractional programming, nonlinear programming, combinatorialalgorithms, etc.).

An objective function defining the high level optimization problem canbe expressed in the linear programming framework as:

${{\underset{x}{argmin}c^{T}x};}{{{{subject}{to}{Ax}} \leq b},{{Hx} = g}}$

where c is a cost vector, x is a decision matrix, A and b are a matrixand vector (respectively) which describe inequality constraints on thevariables in the decision matrix x, and H and g are a matrix and vector(respectively) which describe equality constraints on the variables inthe decision matrix x. The variables in the decision matrix x mayinclude the subplant loads assigned to the various subplants and/or anamount of resource consumption by the subplants at each time element inthe optimization period. The high level optimization circuit may definethe cost vector c and the optimization constraints (e.g., the matrices Aand H and the vectors b and g) and solve the optimization problem todetermine optimal subplant load values for the variables in the decisionmatrix x.

The high level optimization circuit may receive, as an input, predictedor planned energy loads for the building or campus for each of the timeelements in the optimization period. The high level optimization circuitmay use the predicted or planned loads to formulate the constraints onthe high level optimization problem (e.g., to define the matrices A andH and the vectors b and g). The high level optimization circuit may alsoreceive utility rates (e.g., energy prices, water prices, demandcharges, etc.) defining the cost of each resource consumed by thecentral plant to serve the energy loads. The utility rates may betime-variable rates (e.g., defining a different rates at differenttimes) and may include demand charges for various time periods. The highlevel optimization circuit may use the utility rates to define the costvector c.

The high level optimization circuit may receive or generate subplantcurves for each of the subplants. A subplant curve defines the resourceconsumption of a subplant as a function of the load produced by thesubplant. The subplant curves may be generated by a low leveloptimization circuit or by the high level optimization circuit based onoperating data points received from the low level optimization circuit.The high level optimization circuit may use the subplant curves toconstrain the resource consumption of each subplant to a value along thecorresponding subplant curve (e.g., based on the load produced by thesubplant). For example, the high level optimization circuit may use thesubplant curves to define the optimization constraints (e.g., thematrices A and H and the vectors b and g) on the high level optimizationproblem.

In some embodiments, the high level optimization circuit is configuredto incorporate a demand charge into the high level optimization process.The demand charge is an additional charge imposed by some utilityproviders based on the maximum rate of resource consumption during anapplicable demand charge period. For example, an electric demand chargemay be provided as a cost c_(demand) per unit power and may bemultiplied by the peak electricity usage max(P_(elec,k)) during a demandcharge period to determine the demand charge. Conventional systems havebeen unable to incorporate a demand charge into a linear optimizationframework due to the nonlinear max( ) function used to calculate thedemand charge.

Advantageously, the high level optimization circuit of the presentdisclosure may be configured to incorporate the demand charge into thelinear optimization framework by modifying the decision matrix x, thecost vector c, and/or the A matrix and the b vector which describe theinequality constraints. For example, the high level optimization circuitmay modify the decision matrix x by adding a new decision variablex_(peak) representing the peak power consumption within the optimizationperiod. The high level optimization circuit may modify the cost vector cwith the demand charge rate c_(demand) such that the demand charge ratec_(demand) is multiplied by the peak power consumption x_(peak). Thehigh level optimization circuit may generate and/or impose constraintsto ensure that the peak power consumption x_(peak) is greater than orequal to the electric demand for each time step in the demand chargeperiod and greater than or equal to its previous value during the demandcharge period.

In some embodiments, the high level optimization circuit is configuredto incorporate a load change penalty into the high level optimizationprocess. The load change penalty may represent an increased cost (e.g.,equipment degradation, etc.) resulting from a rapid change in the loadassigned to a subplant. The high level optimization circuit mayincorporate the load change penalty by modifying the decision matrix x,the cost vector c, and/or the optimization constraints. For example, thehigh level optimization circuit may modify the decision matrix x byadding load change variables δ for each subplant. The load changevariables may represent the change in subplant load for each subplantfrom one time element to the next. The high level optimization circuitmay modify the cost vector c to add a cost associated with changing thesubplant loads. In some embodiments, the high level optimization circuitadds constraints that constrain the load change variables δ to thecorresponding change in the subplant load. These and other enhancementsto the high level optimization process may be incorporated into thelinear optimization framework, as described in greater detail below.

Central Plant Optimization

Referring now to FIG. 1 , a diagram of a central plant 10 is shown,according to an exemplary embodiment. The central plant 10 is one typeof system that may be used to heat or cool a data center. In someembodiments, an HVAC system is used instead or in addition to thecentral plant 10. Central plant 10 is shown to include a plurality ofsubplants including a heater subplant 12, a heat recovery chillersubplant 14, a chiller subplant 16, a cooling tower subplant 18, a hotthermal energy storage (TES) subplant 20, and a cold thermal energystorage (TES) subplant 22. Subplants 12-22 consume resources (e.g.,water, natural gas, electricity, etc.) from utilities to serve thethermal energy loads (e.g., hot water, cold water, heating, cooling,etc.) of a building or campus. For example, heater subplant 12 may beconfigured to heat water in a hot water loop 24 that circulates the hotwater between central plant 10 and a building (not shown). Chillersubplant 16 may be configured to chill water in a cold water loop 26that circulates the cold water between central plant 10 and thebuilding. Heat recovery chiller subplant 14 may be configured totransfer heat from cold water loop 26 to hot water loop 24 to provideadditional heating for the hot water and additional cooling for the coldwater. Condenser water loop 28 may absorb heat from the cold water inchiller subplant 16 and reject the absorbed heat in cooling towersubplant 18 or transfer the absorbed heat to hot water loop 24. Hot TESsubplant 20 and cold TES subplant 22 store hot and cold thermal energy,respectively, for subsequent use.

Hot water loop 24 and cold water loop 26 may deliver the heated and/orchilled water to air handlers located on the rooftop of a building or toindividual floors or zones of the building. The air handlers push airpast heat exchangers (e.g., heating coils or cooling coils) throughwhich the water flows to provide heating or cooling for the air. Theheated or cooled air may be delivered to individual zones of thebuilding to serve the thermal energy loads of the building. The waterthen returns to central plant 10 to receive further heating or coolingin subsystems 12-22.

Although central plant 10 is shown and described as heating and coolingwater for circulation to a building, it is understood that any othertype of working fluid (e.g., glycol, CO2, etc.) may be used in place ofor in addition to water to serve the thermal energy loads. In otherembodiments, central plant 10 may provide heating and/or coolingdirectly to the building or campus without requiring an intermediateheat transfer fluid. Central plant 10 may be physically separate from abuilding served by subplants 12-22 or physically integrated with thebuilding (e.g., located within the building).

Each of subplants 12-22 may include a variety of equipment configured tofacilitate the functions of the subplant. For example, heater subplant12 is shown to include a plurality of heating elements 30 (e.g.,boilers, electric heaters, etc.) configured to add heat to the hot waterin hot water loop 24. Heater subplant 12 is also shown to includeseveral pumps 32 and 34 configured to circulate the hot water in hotwater loop 24 and to control the flow rate of the hot water throughindividual heating elements 30. Heat recovery chiller subplant 14 isshown to include a plurality of heat recovery heat exchangers 36 (e.g.,refrigeration circuits) configured to transfer heat from cold water loop26 to hot water loop 24. Heat recovery chiller subplant 14 is also shownto include several pumps 38 and 40 configured to circulate the hot waterand/or cold water through heat recovery heat exchangers 36 and tocontrol the flow rate of the water through individual heat recovery heatexchangers 36.

Chiller subplant 16 is shown to include a plurality of chillers 42configured to remove heat from the cold water in cold water loop 26.Chiller subplant 16 is also shown to include several pumps 44 and 46configured to circulate the cold water in cold water loop 26 and tocontrol the flow rate of the cold water through individual chillers 42.Cooling tower subplant 18 is shown to include a plurality of coolingtowers 48 configured to remove heat from the condenser water incondenser water loop 28. Cooling tower subplant 18 is also shown toinclude several pumps 50 configured to circulate the condenser water incondenser water loop 28 and to control the flow rate of the condenserwater through individual cooling towers 48.

Hot TES subplant 20 is shown to include a hot TES tank 52 configured tostore the hot water for later use. Hot TES subplant 20 may also includeone or more pumps or valves configured to control the flow rate of thehot water into or out of hot TES tank 52. Cold TES subplant 22 is shownto include cold TES tanks 54 configured to store the cold water forlater use. Cold TES subplant 22 may also include one or more pumps orvalves configured to control the flow rate of the cold water into or outof cold TES tanks 54. In some embodiments, one or more of the pumps incentral plant 10 (e.g., pumps 32, 34, 38, 40, 44, 46, and/or 50) orpipelines in central plant 10 includes an isolation valve associatedtherewith. In various embodiments, isolation valves may be integratedwith the pumps or positioned upstream or downstream of the pumps tocontrol the fluid flows in central plant 10. In other embodiments, more,fewer, or different types of devices may be included in central plant10.

Referring now to FIG. 2 , a block diagram illustrating a central plantsystem 100 is shown, according to an exemplary embodiment. System 100 isshown to include a central plant controller 102, a building automationsystem 108, and a plurality of subplants 12-22. Subplants 12-22 may bethe same as previously described with reference to FIG. 1 . For example,subplants 12-22 are shown to include a heater subplant 12, a heatrecovery chiller subplant 14, a chiller subplant 16, a hot TES subplant20, and a cold TES subplant 22.

Each of subplants 12-22 is shown to include equipment 60 that can becontrolled by central plant controller 102 and/or building automationsystem 108 to optimize the performance of central plant 10. Equipment 60may include, for example, heating devices 30, chillers 42, heat recoveryheat exchangers 36, cooling towers 48, thermal energy storage devices52, 54, pumps 32, 44, 50, valves 34, 38, 46, and/or other devices ofsubplants 12-22. Individual devices of equipment 60 can be turned on oroff to adjust the thermal energy load served by each of subplants 12-22.In some embodiments, individual devices of equipment 60 can be operatedat variable capacities (e.g., operating a chiller at 10% capacity or 60%capacity) according to an operating setpoint received from central plantcontroller 102.

In some embodiments, one or more of subplants 12-22 includes a subplantlevel controller configured to control the equipment 60 of thecorresponding subplant. For example, central plant controller 102 maydetermine an on/off configuration and global operating setpoints forequipment 60. In response to the on/off configuration and receivedglobal operating setpoints, the subplant controllers may turn individualdevices of equipment 60 on or off, and implement specific operatingsetpoints (e.g., damper position, vane position, fan speed, pump speed,etc.) to reach or maintain the global operating setpoints.

Building automation system (BAS) 108 may be configured to monitorconditions within a controlled building or building zone. For example,BAS 108 may receive input from various sensors (e.g., temperaturesensors, humidity sensors, airflow sensors, voltage sensors, etc.)distributed throughout the building and may report building conditionsto central plant controller 102. Building conditions may include, forexample, a temperature of the building or a zone of the building, apower consumption (e.g., electric load) of the building, a state of oneor more actuators configured to affect a controlled state within thebuilding, or other types of information relating to the controlledbuilding. BAS 108 may operate subplants 12-22 to affect the monitoredconditions within the building and to serve the thermal energy loads ofthe building.

BAS 108 may receive control signals from central plant controller 102specifying on/off states and/or setpoints for equipment 60. BAS 108 maycontrol equipment 60 (e.g., via actuators, power relays, etc.) inaccordance with the control signals provided by central plant controller102. For example, BAS 108 may operate equipment 60 using closed loopcontrol to achieve the setpoints specified by central plant controller102. In various embodiments, BAS 108 may be combined with central plantcontroller 102 or may be part of a separate building management system.According to an exemplary embodiment, BAS 108 is a METASYS® brandbuilding management system, as sold by Johnson Controls, Inc.

Central plant controller 102 may monitor the status of the controlledbuilding using information received from BAS 108. Central plantcontroller 102 may be configured to predict the thermal energy loads(e.g., heating loads, cooling loads, etc.) of the building for pluralityof time steps in a prediction window (e.g., using weather forecasts froma weather service). Central plant controller 102 may generate on/offdecisions and/or setpoints for equipment 60 to minimize the cost ofenergy consumed by subplants 12-22 to serve the predicted heating and/orcooling loads for the duration of the prediction window. Central plantcontroller 102 may be configured to carry out process 1100 (FIG. 11 )and other processes described herein. According to an exemplaryembodiment, central plant controller 102 is integrated within a singlecomputer (e.g., one server, one housing, etc.). In various otherexemplary embodiments, central plant controller 102 can be distributedacross multiple servers or computers (e.g., that can exist indistributed locations). In another exemplary embodiment, central plantcontroller 102 may integrated with a smart building manager that managesmultiple building systems and/or combined with BAS 108.

Central plant controller 102 is shown to include a communicationsinterface 104 and a processing circuit 106. Communications interface 104may include wired or wireless interfaces (e.g., jacks, antennas,transmitters, receivers, transceivers, wire terminals, etc.) forconducting data communications with various systems, devices, ornetworks. For example, communications interface 104 may include anEthernet card and port for sending and receiving data via anEthernet-based communications network and/or a WiFi transceiver forcommunicating via a wireless communications network. Communicationsinterface 104 may be configured to communicate via local area networksor wide area networks (e.g., the Internet, a building WAN, etc.) and mayuse a variety of communications protocols (e.g., BACnet, IP, LON, etc.).

Communications interface 104 may be a network interface configured tofacilitate electronic data communications between central plantcontroller 102 and various external systems or devices (e.g., BAS 108,subplants 12-22, etc.). For example, central plant controller 102 mayreceive information from BAS 108 indicating one or more measured statesof the controlled building (e.g., temperature, humidity, electric loads,etc.) and one or more states of subplants 12-22 (e.g., equipment status,power consumption, equipment availability, etc.). Communicationsinterface 104 may receive inputs from BAS 108 and/or subplants 12-22 andmay provide operating parameters (e.g., on/off decisions, setpoints,etc.) to subplants 12-22 via BAS 108. The operating parameters may causesubplants 12-22 to activate, deactivate, or adjust a setpoint forvarious devices of equipment 60.

Still referring to FIG. 2 , processing circuit 106 is shown to include aprocessor 110 and memory 112. Processor 110 may be a general purpose orspecific purpose processor, an application specific integrated circuit(ASIC), one or more field programmable gate arrays (FPGAs), a group ofprocessing components, or other suitable processing components.Processor 110 may be configured to execute computer code or instructionsstored in memory 112 or received from other computer readable media(e.g., CDROM, network storage, a remote server, etc.).

Memory 112 may include one or more devices (e.g., memory units, memorydevices, storage devices, etc.) for storing data and/or computer codefor completing and/or facilitating the various processes described inthe present disclosure. Memory 112 may include random access memory(RAM), read-only memory (ROM), hard drive storage, temporary storage,non-volatile memory, flash memory, optical memory, or any other suitablememory for storing software objects and/or computer instructions. Memory112 may include database components, object code components, scriptcomponents, or any other type of information structure for supportingthe various activities and information structures described in thepresent disclosure. Memory 112 may be communicably connected toprocessor 110 via processing circuit 106 and may include computer codefor executing (e.g., by processor 106) one or more processes describedherein.

Still referring to FIG. 2 , memory 112 is shown to include a buildingstatus monitor 134. Central plant controller 102 may receive dataregarding the overall building or building space to be heated or cooledwith central plant 10 via building status monitor 134. In an exemplaryembodiment, building status monitor 134 may include a graphical userinterface component configured to provide graphical user interfaces to auser for selecting building requirements (e.g., overall temperatureparameters, selecting schedules for the building, selecting differenttemperature levels for different building zones, etc.).

Central plant controller 102 may determine on/off configurations andoperating setpoints to satisfy the building requirements received frombuilding status monitor 134. In some embodiments, building statusmonitor 134 receives, collects, stores, and/or transmits cooling loadrequirements, building temperature setpoints, occupancy data, weatherdata, energy data, schedule data, and other building parameters. In someembodiments, building status monitor 134 stores data regarding energycosts, such as pricing information available from utilities 126 (energycharge, demand charge, etc.).

Still referring to FIG. 2 , memory 112 is shown to include a load/rateprediction circuit 122. Load/rate prediction circuit 122 may beconfigured to predict the thermal energy loads (

_(k)) of the building or campus for each time step k (e.g., k=1 n) of anoptimization period. Load/rate prediction circuit 122 is shown receivingweather forecasts from a weather service 124. In some embodiments,load/rate prediction circuit 122 predicts the thermal energy loads

_(k) as a function of the weather forecasts. In some embodiments,load/rate prediction circuit 122 uses feedback from BAS 108 to predictloads

_(k). Feedback from BAS 108 may include various types of sensory inputs(e.g., temperature, flow, humidity, enthalpy, etc.) or other datarelating to the controlled building (e.g., inputs from a HVAC system, alighting control system, a security system, a water system, etc.).

In an embodiment where the central plant and/or an HVAC system serves adata center, the load/rate prediction circuit 122 receives a predictionof heat generated by data center equipment in the data center from adata equipment heat predictor circuit 300. The data equipment heatpredictor circuit 300 is described in more detail with reference to FIG.12 . Because a significant amount of heat may be generated by theoperation of data center equipment in a data center, the heat from thedata center equipment may substantially alter the load on the HVACsystem and/or central plant that is configured to maintain the datacenter at or around a desired temperature setpoint. Accordingly, theload/rate predictor 300 is configured to incorporate the predicted heatgenerated by the data center equipment from the data equipment heatpredictor circuit 300 in generating the thermal energy loads (

_(k)) of the building or campus for each time step k of the optimizationperiod. The predicted heat generated by the data center equipment fromthe data equipment heat predictor circuit 300 may thereby be propagatedthrough and influence the operation of the central plant controller 102including the high level optimization circuit 130 and the low leveloptimization circuit 132 to cause the subplants 12-22 to be controlledbased in part on the predicted heat generated by the data centerequipment.

In some embodiments, load/rate prediction circuit 122 receives ameasured electric load and/or previous measured load data from BAS 108(e.g., via building status monitor 134). Load/rate prediction circuit122 may predict loads

_(k) as a function of a given weather forecast ({circumflex over(ϕ)}_(w)), a day type (day), the time of day (t), and previous measuredload data (Y_(k−1)). Such a relationship is expressed in the followingequation:

_(k) =f({circumflex over (ϕ)}_(w) ,day,t|Y _(k−1))

In some embodiments, load/rate prediction circuit 122 uses adeterministic plus stochastic model trained from historical load data topredict loads 4. Load/rate prediction circuit 122 may use any of avariety of prediction methods to predict loads

_(k) (e.g., linear regression for the deterministic portion and an ARmodel for the stochastic portion). Load/rate prediction circuit 122 maypredict one or more different types of loads for the building or campus.For example, load/rate prediction circuit 122 may predict a hot waterload

_(Hot,k) and a cold water load

_(Cold,k) for each time step k within the prediction window.

Load/rate prediction circuit 122 is shown receiving utility rates fromutilities 126. Utility rates may indicate a cost or price per unit of aresource (e.g., electricity, natural gas, water, etc.) provided byutilities 126 at each time step k in the prediction window. In someembodiments, the utility rates are time-variable rates. For example, theprice of electricity may be higher at certain times of day or days ofthe week (e.g., during high demand periods) and lower at other times ofday or days of the week (e.g., during low demand periods). The utilityrates may define various time periods and a cost per unit of a resourceduring each time period. Utility rates may be actual rates received fromutilities 126 or predicted utility rates estimated by load/rateprediction circuit 122.

In some embodiments, the utility rates include demand charges for one ormore resources provided by utilities 126. A demand charge may define aseparate cost imposed by utilities 126 based on the maximum usage of aparticular resource (e.g., maximum energy consumption) during a demandcharge period. The utility rates may define various demand chargeperiods and one or more demand charges associated with each demandcharge period. In some instances, demand charge periods may overlappartially or completely with each other and/or with the predictionwindow. Advantageously, optimization circuit 128 may be configured toaccount for demand charges in the high level optimization processperformed by high level optimization circuit 130. Utilities 126 may bedefined by time-variable (e.g., hourly) prices, a maximum service level(e.g., a maximum rate of consumption allowed by the physicalinfrastructure or by contract) and, in the case of electricity, a demandcharge or a charge for the peak rate of consumption within a certainperiod.

Load/rate prediction circuit 122 may store the predicted loads

_(k) and the utility rates in memory 112 and/or provide the predictedloads

_(k) and the utility rates to optimization circuit 128. Optimizationcircuit 128 may use the predicted loads

_(k) and the utility rates to determine an optimal load distribution forsubplants 12-22 and to generate on/off decisions and setpoints forequipment 60.

Still referring to FIG. 2 , memory 112 is shown to include anoptimization circuit 128. Optimization circuit 128 may perform acascaded optimization process to optimize the performance of centralplant 10. For example, optimization circuit 128 is shown to include ahigh level optimization circuit 130 and a low level optimization circuit132. High level optimization circuit 130 may control an outer (e.g.,subplant level) loop of the cascaded optimization. High leveloptimization circuit 130 may determine an optimal distribution ofthermal energy loads across subplants 12-22 for each time step in theprediction window in order to optimize (e.g., minimize) the cost ofenergy consumed by subplants 12-22. Low level optimization circuit 132may control an inner (e.g., equipment level) loop of the cascadedoptimization. Low level optimization circuit 132 may determine how tobest run each subplant at the load setpoint determined by high leveloptimization circuit 130. For example, low level optimization circuit132 may determine on/off states and/or operating setpoints for variousdevices of equipment 60 in order to optimize (e.g., minimize) the energyconsumption of each subplant while meeting the thermal energy loadsetpoint for the subplant. The cascaded optimization process isdescribed in greater detail with reference to FIG. 3 .

Still referring to FIG. 2 , memory 112 is shown to include a subplantcontrol circuit 138. Subplant control circuit 138 may store historicaldata regarding past operating statuses, past operating setpoints, andinstructions for calculating and/or implementing control parameters forsubplants 12-22. Subplant control circuit 138 may also receive, store,and/or transmit data regarding the conditions of individual devices ofequipment 60, such as operating efficiency, equipment degradation, adate since last service, a lifespan parameter, a condition grade, orother device-specific data. Subplant control circuit 138 may receivedata from subplants 12-22 and/or BAS 108 via communications interface104. Subplant control circuit 138 may also receive and store on/offstatuses and operating setpoints from low level optimization circuit132.

Data and processing results from optimization circuit 128, subplantcontrol circuit 138, or other circuits of central plant controller 102may be accessed by (or pushed to) monitoring and reporting applications136. Monitoring and reporting applications 136 may be configured togenerate real time “system health” dashboards that can be viewed andnavigated by a user (e.g., a central plant engineer). For example,monitoring and reporting applications 136 may include a web-basedmonitoring application with several graphical user interface (GUI)elements (e.g., widgets, dashboard controls, windows, etc.) fordisplaying key performance indicators (KPI) or other information tousers of a GUI. In addition, the GUI elements may summarize relativeenergy use and intensity across central plants in different buildings(real or modeled), different campuses, or the like. Other GUI elementsor reports may be generated and shown based on available data that allowusers to assess performance across one or more central plants from onescreen. The user interface or report (or underlying data engine) may beconfigured to aggregate and categorize operating conditions by building,building type, equipment type, and the like. The GUI elements mayinclude charts or histograms that allow the user to visually analyze theoperating parameters and power consumption for the devices of thecentral plant.

Still referring to FIG. 2 , central plant controller 102 may include oneor more GUI servers, web services 114, or GUI engines 116 to supportmonitoring and reporting applications 136. In various embodiments,applications 136, web services 114, and GUI engine 116 may be providedas separate components outside of central plant controller 102 (e.g., aspart of a smart building manager). Central plant controller 102 may beconfigured to maintain detailed historical databases (e.g., relationaldatabases, XML, databases, etc.) of relevant data and includes computercode circuits that continuously, frequently, or infrequently query,aggregate, transform, search, or otherwise process the data maintainedin the detailed databases. Central plant controller 102 may beconfigured to provide the results of any such processing to otherdatabases, tables, XML files, or other data structures for furtherquerying, calculation, or access by, for example, external monitoringand reporting applications.

Central plant controller 102 is shown to include configuration tools118. Configuration tools 118 can allow a user to define (e.g., viagraphical user interfaces, via prompt-driven “wizards,” etc.) howcentral plant controller 102 should react to changing conditions in thecentral plant subsystems. In an exemplary embodiment, configurationtools 118 allow a user to build and store condition-response scenariosthat can cross multiple central plant devices, multiple buildingsystems, and multiple enterprise control applications (e.g., work ordermanagement system applications, entity resource planning applications,etc.). For example, configuration tools 118 can provide the user withthe ability to combine data (e.g., from subsystems, from eventhistories) using a variety of conditional logic. In varying exemplaryembodiments, the conditional logic can range from simple logicaloperators between conditions (e.g., AND, OR, XOR, etc.) to pseudo-codeconstructs or complex programming language functions (allowing for morecomplex interactions, conditional statements, loops, etc.).Configuration tools 118 can present user interfaces for building suchconditional logic. The user interfaces may allow users to definepolicies and responses graphically. In some embodiments, the userinterfaces may allow a user to select a pre-stored or pre-constructedpolicy and adapt it or enable it for use with their system.

Referring now to FIG. 3 , a block diagram illustrating a portion ofcentral plant system 100 in greater detail is shown, according to anexemplary embodiment. FIG. 3 illustrates the cascaded optimizationprocess performed by optimization circuit 128 to optimize theperformance of central plant 10. In the cascaded optimization process,high level optimization circuit 130 performs a subplant leveloptimization that determines an optimal distribution of thermal energyloads across subplants 12-22 for each time step in the prediction windowin order to minimize the cost of energy consumed by subplants 12-22. Lowlevel optimization circuit 132 performs an equipment level optimizationthat determines how to best run each subplant at the subplant loadsetpoint determined by high level optimization circuit 130. For example,low level optimization circuit 132 may determine on/off states and/oroperating setpoints for various devices of equipment 60 in order tooptimize the energy consumption of each subplant while meeting thethermal energy load setpoint for the subplant.

One advantage of the cascaded optimization process performed byoptimization circuit 128 is the optimal use of computational time. Forexample, the subplant level optimization performed by high leveloptimization circuit 130 may use a relatively long time horizon due tothe operation of the thermal energy storage. However, the equipmentlevel optimization performed by low level optimization circuit 132 mayuse a much shorter time horizon or no time horizon at all since the lowlevel system dynamics are relatively fast (compared to the dynamics ofthe thermal energy storage) and the low level control of equipment 60may be handled by BAS 108. Such an optimal use of computational timemakes it possible for optimization circuit 128 to perform the centralplant optimization in a short amount of time, allowing for real-timepredictive control. For example, the short computational time enablesoptimization circuit 128 to be implemented in a real-time planning toolwith interactive feedback.

Another advantage of the cascaded optimization performed by optimizationcircuit 128 is that the central plant optimization problem can be splitinto two cascaded subproblems. The cascaded configuration provides alayer of abstraction that allows high level optimization circuit 130 todistribute the thermal energy loads across subplants 12-22 withoutrequiring high level optimization circuit 130 to know or use any detailsregarding the particular equipment configuration within each subplant.The interconnections between equipment 60 within each subplant may behidden from high level optimization circuit 130 and handled by low leveloptimization circuit 132. For purposes of the subplant leveloptimization performed by high level optimization circuit 130, eachsubplant may be completely defined by one or more subplant curves 140.

Still referring to FIG. 3 , low level optimization circuit 132 maygenerate and provide subplant curves 140 to high level optimizationcircuit 130. Subplant curves 140 may indicate the rate of utility use byeach of subplants 12-22 (e.g., electricity use measured in kW, water usemeasured in L/s, etc.) as a function of the subplant load. Exemplarysubplant curves are shown and described in greater detail with referenceto FIGS. 5A-8 . In some embodiments, low level optimization circuit 132generates subplant curves 140 based on equipment models 120 (e.g., bycombining equipment models 120 for individual devices into an aggregatecurve for the subplant). Low level optimization circuit 132 may generatesubplant curves 140 by running the low level optimization process forseveral different loads and weather conditions to generate multiple datapoints. Low level optimization circuit 132 may fit a curve to the datapoints to generate subplant curves 140. In other embodiments, low leveloptimization circuit 132 provides the data points to high leveloptimization circuit 132 and high level optimization circuit 132generates the subplant curves using the data points.

High level optimization circuit 130 may receive the load and ratepredictions from load/rate prediction circuit 122 and the subplantcurves 140 from low level optimization circuit 132. The load predictionsmay be based on weather forecasts from weather service 124 and/orinformation from building automation system 108 (e.g., a currentelectric load of the building, measurements from the building, a historyof previous loads, a setpoint trajectory, etc.). The utility ratepredictions may be based on utility rates received from utilities 126and/or utility prices from another data source. High level optimizationcircuit 130 may determine the optimal load distribution for subplants12-22 (e.g., a subplant load for each subplant) for each time step theprediction window and provide the subplant loads as setpoints to lowlevel optimization circuit 132. In some embodiments, high leveloptimization circuit 130 determines the subplant loads by minimizing thetotal operating cost of central plant 10 over the prediction window. Inother words, given a predicted load and utility rate information fromload/rate prediction circuit 122, high level optimization circuit 130may distribute the predicted load across subplants 12-22 over theoptimization period to minimize operating cost.

In some instances, the optimal load distribution may include using TESsubplants 20 and/or 22 to store thermal energy during a first time stepfor use during a later time step. Thermal energy storage mayadvantageously allow thermal energy to be produced and stored during afirst time period when energy prices are relatively low and subsequentlyretrieved and used during a second time period when energy proves arerelatively high. The high level optimization may be different from thelow level optimization in that the high level optimization has a longertime constant due to the thermal energy storage provided by TESsubplants 20-22. The high level optimization may be described by thefollowing equation:

$\theta_{HL}^{*} = {\arg\min\limits_{\theta_{HL}}{J_{HL}\left( \theta_{HL} \right)}}$

where θ*_(HL) contains the optimal high level decisions (e.g., theoptimal load for each of subplants 12-22) for the entire optimizationperiod and J_(HL) is the high level cost function.

To find the optimal high level decisions θ*_(HL), high leveloptimization circuit 132 may minimize the high level cost functionJ_(HL). The high level cost function J_(HL) may be the sum of theeconomic costs of each utility consumed by each of subplants 12-22 forthe duration of the optimization period. In some embodiments, the highlevel cost function J_(HL) may be described using the followingequation:

${J_{HL}\left( \theta_{HL} \right)} = {\sum\limits_{k = 1}^{n_{h}}{\sum\limits_{i = 1}^{n_{s}}\left\lbrack {\sum\limits_{j = 1}^{n_{u}}{{t_{s} \cdot c_{jk}}{u_{jik}\left( \theta_{HL} \right)}}} \right\rbrack}}$

where n_(h) is the number of time steps k in the optimization period,n_(s) is the number of subplants, t_(s) is the duration of a time step,c_(jk) is the economic cost of utility j at a time step k of theoptimization period, and u_(jik) is the rate of use of utility j bysubplant i at time step k.

In some embodiments, the cost function J_(HL) includes an additionaldemand charge term such as:

$w_{d}c_{demand}\underset{n_{h}}{\max}\left( {{u_{elec}\left( \theta_{HL} \right)},u_{\max,{ele}}} \right)$

where w_(d) is a weighting term, c_(demand) is the demand cost, and themax( ) term selects the peak electricity use during the applicabledemand charge period. Accordingly, the high level cost function J_(HL)may be described by the equation:

${J_{HL}\left( \theta_{HL} \right)} = {{\sum\limits_{k = 1}^{n_{h}}{\sum\limits_{i = 1}^{n_{s}}\left\lbrack {\sum\limits_{j = 1}^{n_{u}}{{t_{s} \cdot c_{jk}}{u_{jik}\left( \theta_{HL} \right)}}} \right\rbrack}} + {w_{d}c_{demand}\underset{n_{h}}{\max}\left( {{u_{elec}\left( \theta_{HL} \right)},u_{\max,{ele}}} \right)}}$

The decision vector θ_(HL) may be subject to several constraints. Forexample, the constraints may require that the subplants not operate atmore than their total capacity, that the thermal storage not charge ordischarge too quickly or under/over flow for the tank, and that thethermal energy loads for the building or campus are met. Theserestrictions lead to both equality and inequality constraints on thehigh level optimization problem, as described in greater detail withreference to FIG. 4 .

Still referring to FIG. 3 , low level optimization circuit 132 may usethe subplant loads determined by high level optimization circuit 130 todetermine optimal low level decisions θ*_(LL) (e.g. binary on/offdecisions, flow setpoints, temperature setpoints, etc.) for equipment60. The low level optimization process may be performed for each ofsubplants 12-22. Low level optimization circuit 132 may be responsiblefor determining which devices of each subplant to use and/or theoperating setpoints for such devices that will achieve the subplant loadsetpoint while minimizing energy consumption. The low level optimizationmay be described using the following equation:

$\theta_{LL}^{*} = {\arg\min\limits_{\theta_{LL}}{J_{LL}\left( \theta_{LL} \right)}}$

where θ*_(LL) contains the optimal low level decisions and J_(LL) is thelow level cost function.

To find the optimal low level decisions θ*_(LL), low level optimizationcircuit 132 may minimize the low level cost function J_(LL). The lowlevel cost function J_(LL) may represent the total energy consumptionfor all of equipment 60 in the applicable subplant. The low level costfunction J_(LL) may be described using the following equation:

${J_{LL}\left( \theta_{LL} \right)} = {\sum\limits_{j = 1}^{N}{t_{s} \cdot b_{j} \cdot {u_{j}\left( \theta_{LL} \right)}}}$

where N is the number of devices of equipment 60 in the subplant, t_(s)is the duration of a time step, b_(j) is a binary on/off decision (e.g.,0=off, 1=on), and u_(j) is the energy used by device j as a function ofthe setpoint θ_(LL). Each device may have continuous variables which canbe changed to determine the lowest possible energy consumption for theoverall input conditions.

Low level optimization circuit 132 may minimize the low level costfunction J_(LL) subject to inequality constraints based on thecapacities of equipment 60 and equality constraints based on energy andmass balances. In some embodiments, the optimal low level decisionsθ*_(LL) are constrained by switching constraints defining a shorthorizon for maintaining a device in an on or off state after a binaryon/off switch. The switching constraints may prevent devices from beingrapidly cycled on and off. In some embodiments, low level optimizationcircuit 132 performs the equipment level optimization withoutconsidering system dynamics. The optimization process may be slow enoughto safely assume that the equipment control has reached itssteady-state. Thus, low level optimization circuit 132 may determine theoptimal low level decisions θ*_(LL) at an instance of time rather thanover a long horizon.

Low level optimization circuit 132 may determine optimum operatingstatuses (e.g., on or off) for a plurality of devices of equipment 60.According to an exemplary embodiment, the on/off combinations may bedetermined using binary optimization and quadratic compensation. Binaryoptimization may minimize a cost function representing the powerconsumption of devices in the applicable subplant. In some embodiments,non-exhaustive (i.e., not all potential combinations of devices areconsidered) binary optimization is used. Quadratic compensation may beused in considering devices whose power consumption is quadratic (andnot linear). Low level optimization circuit 132 may also determineoptimum operating setpoints for equipment using nonlinear optimization.Nonlinear optimization may identify operating setpoints that furtherminimize the low level cost function J_(LL). Low level optimizationcircuit 132 may provide the on/off decisions and setpoints to buildingautomation system 108 for use in controlling the central plant equipment60.

In some embodiments, the low level optimization performed by low leveloptimization circuit 132 is the same or similar to the low leveloptimization process described in U.S. patent application Ser. No.14/634,615, filed Feb. 27, 2015, incorporated by reference herein in itsentirety.

Referring now to FIG. 4 , a block diagram illustrating high leveloptimization circuit 130 in greater detail is shown, according to anexemplary embodiment. High level optimization circuit 130 may receiveload and rate predictions from load/rate prediction circuit 122 andsubplant curves from low level optimization circuit 132. High leveloptimization circuit 130 may determine optimal subplant loads for eachof subplants 12-22 as a function of the load and rate predictions andthe subplant curves. In some embodiments, the optimal subplant loadsminimize the economic cost of operating subplants 12-22 to satisfy thepredicted loads for the building or campus. High level optimizationcircuit 130 may output the optimal subplant loads to low leveloptimization circuit 132.

High level optimization circuit 130 is shown to include an optimizationframework circuit 142. Optimization framework circuit 142 may beconfigured to select and/or establish an optimization framework for usein calculating the optimal subplant loads. In some embodiments,optimization framework circuit 142 uses linear programming as theoptimization framework. A linear programming problem has the followingform:

${{\underset{x}{argmin}c^{T}x};}{{{{subject}{to}{Ax}} \leq b},{{Hx} = g}}$

where c is a cost vector, x is a decision matrix, A and b are a matrixand vector (respectively) which describe inequality constraints on theoptimization problem, and H and g are a matrix and vector (respectively)which describe equality constraints on the optimization problem.

The following paragraphs describe an exemplary linear optimizationframework that may be used by high level optimization circuit 130 tocalculate the optimal subplant loads. Advantageously, the linearprogramming framework described herein allows high level optimizationcircuit 130 to determine the subplant load distribution for a longoptimization period in a very short timeframe complete with load changepenalties, demand charges, and subplant performance curves. However, thelinear optimization framework is merely one example of an optimizationframework that can be used by high level optimization circuit 130 andshould not be regarded as limiting. It should be understood that inother embodiments, high level optimization circuit 130 may use any of avariety of other optimization frameworks and/or optimization techniques(e.g., quadratic programming, linear-fractional programming, nonlinearprogramming, combinatorial algorithms, etc.) to calculate the optimalsubplant loads.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a linear program circuit 144. Linear program circuit 144 maybe configured to formulate and solve a linear optimization problem tocalculate the optimal subplant loads. For example, linear programcircuit 144 may determine and set values for the cost vector c, the Amatrix and the b vector which describe the inequality constraints, andthe H matrix and the g vector which describe the equality constraints.Linear program circuit 144 may determine an optimal decision matrix x*that minimizes the cost function c^(T)x. The optimal decision matrix x*may correspond to the optimal decisions θ*_(HL). (for each time step kwithin an optimization period) that minimize the high level costfunction J_(HL), as described with reference to FIG. 3 .

For a central plant 10 that includes chillers, heat recovery chillers,hot water generators, and thermal energy storage, the plant assetsacross which the loads are to be distributed may include a chillersubplant 16, a heat recovery chiller subplant 14, a heater subplant 12,a hot thermal energy storage subplant 20, and a cold thermal energystorage subplant 22. The loads across each of subplants 12-22 may be thedecision variables in the decision matrix x that the high leveloptimization determines for each time step k within the optimizationperiod. For example, linear program circuit 144 may formulate thedecision matrix x as:

x=8 {dot over (Q)} _(chiller,1 . . . n) ,{dot over (Q)}_(hrChiller,1 . . . n) ,{dot over (Q)} _(Heater,1 . . . n) ,{dot over(Q)} _(HotStorage,1 . . . n) ,{dot over (Q)} _(ColdStorage,1 . . . n)^(T)

where {dot over (Q)}_(Chiller,1 . . . n), {dot over(Q)}_(hrChiller,1 . . . n), {dot over (Q)}_(Heater,1 . . . n), {dot over(Q)}_(HotStorage,1 . . . n), and {dot over (Q)}_(ColdStorage,1 . . . n)are n-dimensional vectors representing the thermal energy load assignedto chiller subplant 16, heat recovery chiller subplant 14, heatersubplant 12, hot TES subplant 20, and cold TES subplant 22,respectively, for each of the n time steps within the optimizationperiod.

Linear program circuit 144 may formulate the linear program for thesimple case where only energy cost and equipment constraints areconsidered. The simplified linear program may then be modified byinequality constraints circuit 146, equality constraints circuit 148,unmet loads circuit 150, ground loop circuit 152, heat exchanger circuit154, demand charge circuit 156, load change penalty circuit 158, tankforced full circuit 160, and/or subplant curves circuit 170 to provideadditional enhancements, described in greater detail below.

In some embodiments, linear program circuit 144 formulates thesimplified linear program using the assumption that each subplant has aspecific cost per unit load. For example, linear program circuit 144 mayassume that each subplant has a constant coefficient of performance(COP) or efficiency for any given time step k. The COP can change overtime and may have a different value for different time steps; however,in the simplest case, the COP for each of subplant is not a function ofthe loading. With this assumption, linear program circuit 144 mayformulate the cost function c as:

$c = {t_{s} \cdot \left\lbrack {\left\lbrack {\sum\limits_{j = 1}^{n_{u}}{c_{j}u_{j,{Chiller}}}} \right\rbrack_{1\ldots h},\left\lbrack {\sum\limits_{\overset{˙}{j} = 1}^{n_{u}}{c_{j}u_{j,{hrChiller}}}} \right\rbrack_{1\ldots h},\left\lbrack {\sum\limits_{j = 1}^{n_{u}}{c_{j}u_{j,{Heater}}}} \right\rbrack_{1\ldots h},0_{h},0_{h}} \right\rbrack^{T}}$

where t_(s) is the duration of a time step, n_(u) is the total number ofresources (e.g., electricity, natural gas, water, etc.) consumed by thesubplants, c_(j) is the cost per unit of the jth resource, andu_(j,Chiller), u_(j,hrChiller), and u_(j,Heater) are the usage rates ofthe jth resource by chiller subplant 16, heat recovery chiller subplant14, and heater subplant 12, respectively, for each of the h time stepswithin the optimization period. The first three elements of the form[Σ_(j−1) ^(n) ^(u) c_(j)u_(j)]_(1 . . . h) represent vectors of h sums(i.e., summing over all resource use), one for each time step within theoptimization period. The last two elements of the form 0_(h) are zero toindicate that charging or discharging the thermal energy storage tankshas no cost (pumping power is neglected).

In some embodiments, linear program circuit 144 uses the load and ratepredictions to formulate the linear program. For example, linear programcircuit 144 may use the load predictions to determine values foru_(j,Chiller), u_(j,hrChiller), and u_(j,Heater) and may use the ratepredictions to determine values for c_(j) for each of the n_(u)resources. In some embodiments, linear program circuit 144 uses thesubplant curves to define c_(j) as a function of the resource usage.Linear program circuit 144 may use inputs from inequality constraintscircuit 146, equality constraints circuit 148, unmet loads circuit 150,ground loop circuit 152, heat exchanger circuit 154, demand chargecircuit 156, load change penalty circuit 158, tank forced full circuit160, and/or subplant curves circuit 170 to determine and set values forthe various matrices and vectors in the linear program. Circuits 146-170may modify the cost vector c, the A matrix, the b vector, the H matrix,and/or the g vector to provide additional enhancements and/orfunctionality to the linear program. The inputs provided by circuits146-170 are described in greater detail below.

Linear program circuit 144 may use any of a variety of linearoptimization techniques to solve the linear optimization problem. Forexample, linear program circuit 144 may use basis exchange algorithms(e.g., simplex, crisscross, etc.), interior point algorithms (e.g.,ellipsoid, projective, path-following, etc.), covering and packingalgorithms, integer programming algorithms (e.g., cutting-plant, branchand bound, branch and cut, branch and price, etc.), or any other type oflinear optimization algorithm or technique to solve the linear programsubject to the optimization constraints. For embodiments in whichnonlinear optimization is used, linear program circuit 144 may use anyof a variety of nonlinear optimization techniques to solve the nonlinearoptimization problem.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include an inequality constraints circuit 146. Inequality constraintscircuit 146 may formulate or define one or more inequality constraintson the optimization problem solved by linear program circuit 144. Insome instances, inequality constraints circuit 146 defines inequalityconstraints on the decision variables {dot over (Q)}_(Chiller,k), {dotover (Q)}_(hrChiller,k), and {dot over (Q)}_(Heater,k) corresponding tothe loads on chiller subplant 16, heat recovery chiller subplant 14, andheater subplant 12, respectively, for each time step k withinoptimization period. For example, each of subplants 12-16 may have twocapacity constraints given by the following equations:

{dot over (Q)} _(i,k) ≤{dot over (Q)} _(imax) ∀k∈horizon

{dot over (Q)} _(i,k)≥0 ∀k∈horizon

where {dot over (Q)}_(i,k) is the load on the ith subplant during timestep k and {dot over (Q)}_(i,max) is the maximum capacity of the ithsubplant. The first capacity constraint requires the load {dot over(Q)}_(i,k) on each of subplants 12-16 to be less than or equal to themaximum capacity {dot over (Q)}_(i,max) of the subplant for each timestep k within the optimization period. The second capacity constraintrequires the load {dot over (Q)}_(i,k) on each of subplants 12-16 to begreater than or equal to zero for each time step k within theoptimization period.

The inequality constraints for chiller subplant 16 can be placed in theform Ax≤b by defining the A matrix and the b vector as follows:

${A = \begin{bmatrix}\left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack {- I_{h}} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{{\overset{˙}{Q}}_{{Chiller},\max}\left\lbrack I_{h} \right\rbrack} \\\left\lbrack 0_{h} \right\rbrack\end{bmatrix}}$

where [I_(h)] represents either an h by h identity matrix or an h by 1ones vector, [O_(h)] represents either an h by h zero matrix or an h by1 zero vector, and {dot over (Q)}_(Chiller,max) is the maximum capacityof chiller subplant 16. Similar inequality constraints for heat recoverychiller subplant 14 and heater subplant 12 can be placed in the formAx≤b by defining the A matrices and the b vectors as follows:

${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack {- I_{h}} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{{\overset{.}{Q}}_{{hrChiller},\max}\left\lbrack I_{h} \right\rbrack} \\\left\lbrack 0_{h} \right\rbrack\end{bmatrix}}$ ${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack {- I_{h}} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{{\overset{.}{Q}}_{{Heater},\max}\left\lbrack I_{h} \right\rbrack} \\\left\lbrack 0_{h} \right\rbrack\end{bmatrix}}$

where {dot over (Q)}_(hrChiller,max) is the maximum capacity of heatrecovery chiller subplant 14 and {dot over (Q)}_(Heater,max) is themaximum capacity of heater subplant 12.

Inequality constraints circuit 146 may formulate or define inequalityconstraints on the decision variables {dot over (Q)}_(HotStorage,k) and{dot over (Q)}_(ColdStorage,k) corresponding to the loads on hot TESsubplant 20 and cold TES subplant 22 for each time step k within theoptimization period. For example, each of subplants 20-22 may have twocapacity constraints given by the following equations:

{dot over (Q)} _(i,k) ≤{dot over (Q)} _(discharge,i,max) ∀k∈horizon

{dot over (Q)} _(i,k) ≤{dot over (Q)} _(charge,i,max) ∀k∈horizon

where {dot over (Q)}_(i,k) is the rate at which ith TES subplant isbeing discharged at time step k, {dot over (Q)}_(discharge,i,max) is themaximum discharge rate of the ith subplant, and {dot over(Q)}_(charge,i,max) is the maximum charge rate of the ith subplant.Positive load values for {dot over (Q)}_(i,k) indicate that the TESsubplant is discharging and negative load values for {dot over(Q)}_(i,k) indicate that the subplant is charging. The first capacityconstraint requires the discharge rate {dot over (Q)}_(i,k) for each ofsubplants 20-22 to be less than or equal to the maximum discharge rate{dot over (Q)}_(discharge,i,max) of the subplant for each time step kwithin the optimization period. The second capacity constraint requiresthe negative discharge rate −{dot over (Q)}_(i,k) (i.e., the chargerate) for each of subplants 20-22 to be less than or equal to themaximum charge rate {dot over (Q)}_(charge,i,max) of the subplant foreach time step k within the optimization period.

The inequality constraints for hot TES subplant 20 can be placed in theform Ax≤b by defining the A matrix and the b vector as follows:

${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack {- I_{h}} \right\rbrack & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{{\overset{.}{Q}}_{{HotDischarge},\max}\left\lbrack I_{h} \right\rbrack} \\{{\overset{.}{Q}}_{{HotCharge},\max}\left\lbrack I_{h} \right\rbrack}\end{bmatrix}}$

where {dot over (Q)}_(HotDischarge,max) is the maximum discharge ratefor hot TES subplant 20 and {dot over (Q)}_(HotCharge,max) is themaximum charge rate for hot TES subplant 20. Similar inequalityconstraints for cold TES subplant 22 can be placed in the form Ax≤b bydefining the A matrix and the b vector as follows:

${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack {- I_{h}} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{{\overset{.}{Q}}_{{ColdDischarge},\max}\left\lbrack I_{h} \right\rbrack} \\{{\overset{.}{Q}}_{{ColdCharge},\max}\left\lbrack I_{h} \right\rbrack}\end{bmatrix}}$

where {dot over (Q)}_(ColdDischarge,max) is the maximum discharge ratefor cold TES subplant 22 and {dot over (Q)}_(ColdCharge,max) is themaximum charge rate for cold TES subplant 22.

Inequality constraints circuit 146 may implement an electrical demandconstraint for the total electrical usage of all the subplants and thebuilding/campus P_(elec,campus). Inequality constraints circuit 146 mayrequire that the total electrical demand be less than or equal to amaximum electrical demand P_(elec,max) by defining the A matrix and theb vector as follows:

A=[u _(elec,Chiller)[I _(h)],u _(elec,hrChiller)[I _(h)],u_(elec,Heater)[I _(h)],0_(n),0_(n)],b=P _(elec,max)[I _(h)]−P_(elec,campus,k)

where u_(elec,Chiller), u_(elec,hrChiller), and u_(elec,Heater) are theelectrical usage values for chiller subplant 16, heat recovery chillersubplant 14, and heater subplant 12, respectively, P_(elec,campus,k) isthe electrical usage of the building/campus at time k, and P_(elec,max)is the maximum total electrical usage for central plant 10 and thebuilding/campus.

Inequality constraints circuit 146 may implement tank capacityconstraints for hot TES subplant 20 and cold TES subplant 22. The tankcapacity constraints may require that each TES tank never charge aboveits maximum capacity or discharge below zero. These physicalrequirements lead to a series of constraints to ensure that the initialtank level Q₀ of each TES tank at the beginning of the optimizationperiod plus all of the charging during time steps 1 to k into theoptimization period is less than or equal to the maximum capacityQ_(max) of the TES tank. A similar constraint may be implemented toensure that the initial tank level Q₀ of each TES tank at the beginningof the optimization period minus all of the discharging during timesteps 1 to k into the optimization period is greater than or equal tozero.

The tank capacity constraints for hot TES subplant 20 can be placed inthe form Ax≤b by defining the A matrix and the b vector as follows:

${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {t_{s}\left\lbrack \Delta_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {t_{s}\left\lbrack {- \Delta_{h}} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{b = \begin{bmatrix}{Q_{0,{Hot}}\left\lbrack I_{h} \right\rbrack} \\{Q_{\max,{Hot}} - {Q_{0,{Hot}}\left\lbrack I_{h} \right\rbrack}}\end{bmatrix}}$

where Q_(0,Hot) is the initial charge level of hot TES subplant 20 atthe beginning of the optimization period, Q_(max,Hot) is the maximumcharge level of hot TES subplant 20, Δ_(h) is a lower triangular matrixof ones, and t_(s) is the duration of a time step. Discharging the tankis represented in the top row of the A matrix as positive flow from thetank and charging the tank is represented in the bottom row of the Amatrix as negative flow from the tank. Similar inequality constraintsfor cold TES subplant 22 can be placed in the form Ax≤b by defining theA matrix and the b vector as follows:

${A = \begin{bmatrix}\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {t_{s}\left\lbrack \Delta_{h} \right\rbrack} \\\left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {t_{s}\left\lbrack {- \Delta_{h}} \right\rbrack}\end{bmatrix}},{b = \begin{bmatrix}{Q_{0,{Cold}}\left\lbrack I_{h} \right\rbrack} \\{Q_{\max,{Cold}} - {Q_{0,{Cold}}\left\lbrack I_{h} \right\rbrack}}\end{bmatrix}}$

where Q_(0,cold) is the initial charge level of cold TES subplant 22 atthe beginning of the optimization period and Q_(max,COld) is the maximumcharge level of cold TES subplant 22.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include an equality constraints circuit 148. Equality constraintscircuit 148 may formulate or define one or more equality constraints onthe optimization problem solved by linear program circuit 144. Theequality constraints may ensure that the predicted thermal energy loadsof the building or campus are satisfied for each time step k in theoptimization period. Equality constraints circuit 148 may formulate anequality constraint for each type of thermal energy load (e.g., hotwater, cold water, etc.) to ensure that the load is satisfied. Theequality constraints may be given by the following equation:

${\sum\limits_{i = 1}^{n_{s}}{\overset{.}{Q}}_{p,i,k}} = {{\hat{\ell}}_{p,k}{\forall{k \in {horizon}}}}$

where {dot over (Q)}_(p,i,k) is the thermal energy load of type p (e.g.,hot water, cold water, etc.) on the ith subplant during time step k,n_(s) is the total number of subplants capable of serving thermal energyload p, and

_(p,k) is the predicted thermal energy load of type p that must besatisfied at time step k. The predicted thermal energy loads may bereceived as load predictions from load/rate prediction circuit 122.

In some embodiments, the predicted thermal energy loads include apredicted hot water thermal energy load

_(Hot,k) and a predicted cold water thermal energy load

_(COld,k) for each time step k. The predicted hot water thermal energyload

_(Hot,k) may be satisfied by the combination of heat recovery chillersubplant 14, heater subplant 12, and hot TES subplant 20. The predictedcold water thermal energy load

_(Cold,k) may be satisfied by the combination of heat recovery chillersubplant 14, chiller subplant 16, and cold TES subplant 22.

The equality constraints can be placed in the form Hx=g by defining theH matrix and the g vector as follows:

${H = \begin{bmatrix}\left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left( {1 + {u_{{elec},{hrChiller}}\left\lbrack I_{h} \right\rbrack}} \right. & \left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack\end{bmatrix}},{{\mathcal{g}} = \begin{bmatrix}{\hat{\ell}}_{{Cold},{1\ldots k}} \\{\hat{\ell}}_{{Hot},{1\ldots k}}\end{bmatrix}}$

where

_(Cold,1 . . . k) and

_(Hot,1 . . . k) are k-dimensional vectors of predicted cold water loadsand predicted hot water loads, respectively, at each of time steps k,and u_(elec,hrChiller) is the electrical consumption of heat recoverychiller subplant 14. For central plants that serve one or moreadditional types of loads, an additional row may be added to the Hmatrix and the g vector to define the equality constraints for eachadditional load served by the central plant.

For this example problem, assuming an optimization period of 72 one-hoursamples, the linear program has 360 decision variables and 1224constraints. However, linear program circuit 144 can solve this linearprogram to determine the optimal subplant load values in less than 200milliseconds using the linear programming framework. Advantageously,this allows high level optimization circuit 130 to determine thesubplant load distribution for a long optimization period in a veryshort timeframe.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include an unmet loads circuit 150. In some instances, the centralplant equipment 60 may not have enough capacity or reserve storage tosatisfy the predicted thermal energy loads, regardless of how thethermal energy loads are distributed across subplants 12-22. In otherwords, the high level optimization problem may have no solution thatsatisfies all of the inequality and equality constraints, even if theapplicable subplants are operated at maximum capacity. Unmet loadscircuit 150 may be configured to modify the high level optimizationproblem to account for this possibility and to allow the high leveloptimization to find the solution that results in the minimal amount ofunmet loads.

In some embodiments, unmet loads circuit 150 modifies the decisionvariable matrix x by introducing a slack variable for each type ofthermal energy load. The slack variables represent an unsatisfied (e.g.,unmet, deferred, etc.) amount of each type of thermal energy load. Forexample, unmet loads circuit 150 may modify the decision variable matrixx as follows:

x=[{dot over (Q)} _(Chiller,1 . . . n) ,{dot over (Q)}_(hrChiller,1 . . . n) ,{dot over (Q)} _(Heater,1 . . . n) ,Q_(HotStorage,1 . . . n) ,{dot over (Q)} _(ColdStorage,1 . . . n) ,Q_(ColdUnmet,1 . . . n) ,Q _(HotUnmet,1 . . . n)]^(T)

and Q_(coldUnmet,1 . . . n) and Q_(HotUnmet,1 . . . n) are n-dimensionalvectors representing a total deferred cold thermal energy load and atotal deferred hot thermal energy load, respectively, at each time stepk within the optimization period. In some embodiments, the decisionvariables Q_(ColdUnmet,1 . . . n) and Q_(HotUnmet,1 . . . n) representtotal deferred loads that have accumulated up to each time step k ratherthan the incremental deferred load at each time step. The total deferredload may be used because any deferred load is likely to increase therequired load during subsequent time steps.

Unmet loads circuit 150 may modify the equality constraints to accountfor any deferred thermal energy loads. The modified equality constraintsmay require that the predicted thermal energy loads are equal to thetotal loads satisfied by subplants 12-22 plus any unsatisfied thermalenergy loads. The modified equality constraints can be placed in theform Hx=g by defining the H matrix and the g vector as follows:

${H = \text{ }\begin{bmatrix}\left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & {\left\lbrack I_{h} \right\rbrack - \left\lbrack D_{- 1} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left( {1 + {u_{{elec},{hrChiller}}\left\lbrack I_{h} \right\rbrack}} \right. & \left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {\left\lbrack I_{h} \right\rbrack - \left\lbrack D_{- 1} \right\rbrack}\end{bmatrix}},{{\mathcal{g}} = \begin{bmatrix}{\hat{\ell}}_{{Cold},{1\ldots k}} \\{\hat{\ell}}_{{Hot},{1\ldots k}}\end{bmatrix}}$

where [D⁻¹] is a lower diagonal matrix of ones.

Unmet loads circuit 150 may modify the cost vector c to associate costvalues with any unmet loads. In some embodiments, unmet loads circuit150 assigns unmet loads a relatively higher cost compared to the costsassociated with other types of loads in the decision variable matrix x.Assigning a large cost to unmet loads ensures that the optimal solutionto the high level optimization problem uses unmet loads only as a lastresort (i.e., when the optimization has no solution without using unmetloads). Accordingly, linear program circuit 144 may avoid using unmetloads if any feasible combination of equipment is capable of satisfyingthe predicted thermal energy loads. In some embodiments, unmet loadscircuit 150 assigns a cost value to unmet loads that allows linearprogram circuit 144 to use unmet loads in the optimal solution even ifthe central plant is capable of satisfying the predicted thermal energyloads. For example, unmet loads circuit 150 may assign a cost value thatallows linear program circuit 144 to use unmet loads if the solutionwithout unmet loads would be prohibitively expensive and/or highlyinefficient.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a subplant curves circuit 170. In the simplest case describedwith reference to linear program circuit 144, it was assumed that theresource consumption of each subplant is a linear function of thethermal energy load produced by the subplant. However, this assumptionmay not be true for some subplant equipment, much less for an entiresubplant. Subplant curves circuit 170 may be configured to modify thehigh level optimization problem to account for subplants that have anonlinear relationship between resource consumption and load production.

Subplant curves circuit 170 is shown to include a subplant curve updater172, a subplant curves database 174, a subplant curve linearizer 176,and a subplant curves incorporator 178. Subplant curve updater 172 maybe configured to request subplant curves for each of subplants 12-22from low level optimization circuit 132. Each subplant curve mayindicate an amount of resource consumption by a particular subplant(e.g., electricity use measured in kW, water use measured in L/s, etc.)as a function of the subplant load. Exemplary subplant curves are shownand described in greater detail with reference to FIGS. 5A-8 .

In some embodiments, low level optimization circuit 132 generates thesubplant curves by running the low level optimization process forvarious combinations of subplant loads and weather conditions togenerate multiple data points. Low level optimization circuit 132 mayfit a curve to the data points to generate the subplant curves andprovide the subplant curves to subplant curve updater 172. In otherembodiments, low level optimization circuit 132 provides the data pointsto subplant curve updater 172 and subplant curve updater 172 generatesthe subplant curves using the data points. Subplant curve updater 172may store the subplant curves in subplant curves database 174 for use inthe high level optimization process.

In some embodiments, the subplant curves are generated by combiningefficiency curves for individual devices of a subplant. A deviceefficiency curve may indicate the amount of resource consumption by thedevice as a function of load. The device efficiency curves may beprovided by a device manufacturer or generated using experimental data.In some embodiments, the device efficiency curves are based on aninitial efficiency curve provided by a device manufacturer and updatedusing experimental data. The device efficiency curves may be stored inequipment models 120. For some devices, the device efficiency curves mayindicate that resource consumption is a U-shaped function of load.Accordingly, when multiple device efficiency curves are combined into asubplant curve for the entire subplant, the resultant subplant curve maybe a wavy curve as shown in FIG. 6 . The waves are caused by a singledevice loading up before it is more efficient to turn on another deviceto satisfy the subplant load.

Subplant curve linearizer 176 may be configured to convert the subplantcurves into convex curves. A convex curve is a curve for which a lineconnecting any two points on the curve is always above or along thecurve (i.e., not below the curve). Convex curves may be advantageous foruse in the high level optimization because they allow for anoptimization process that is less computationally expensive relative toan optimization process that uses non-convex functions. Subplant curvelinearizer 176 may be configured to break the subplant curves intopiecewise linear segments that combine to form a piecewise-definedconvex curve. An unmodified subplant curve 600 and a linearized subplantcurve 700 generated by subplant curve linearizer 176 are shown in FIGS.6 and 7 , respectively. Subplant curve linearizer 176 may store thelinearized subplant curves in subplant curves database 174.

Still referring to FIG. 4 , subplant curves circuit 170 is shown toinclude a subplant curve incorporator 178. Subplant curve incorporator178 may be configured to modify the high level optimization problem toincorporate the subplant curves into the optimization. In someembodiments, subplant curve incorporator 178 modifies the decisionmatrix x to include one or more decision vectors representing theresource consumption of each subplant. For example, for chiller subplant16, subplant curve incorporator 178 may modify the decision matrix x asfollows:

x=[ . . . Q _(Chiller,1 . . . n) . . . u _(Chiller,elec,1 . . . n) u_(Chiller,water,1 . . . n) . . . ]^(T)

where u_(Chiller,elec,1 . . . n) and u_(Chiller,water,1 . . . n) aren-dimensional vectors representing the amount of electrical consumptionand water consumption, respectively, by chiller subplant 16 at each timestep k.

Subplant curve incorporator 178 may add one or more resource consumptionvectors to matrix x for each of subplants 12-22. The decision vectorsadded by subplant curve incorporator 178 for a given subplant mayrepresent an amount of resource consumption for each resource consumedby the subplant (e.g., water, electricity, natural gas, etc.) at eachtime step k within the optimization period. For example, if heatersubplant 12 consumes natural gas, electricity, and water, subplant curveincorporator 178 may add a decision vector u_(Heater,gas,1 . . . n)representing an amount of natural gas consumed by heater subplant 12 ateach time step, a decision vector u_(Heater,elec,1 . . . n) representingan amount of electricity consumed by heater subplant 12 at each timestep, and a decision vector u_(Heater,water,1 . . . n) representing anamount of water consumed by heater subplant at each time step. Subplantcurve incorporator 178 may add resource consumption vectors for othersubplants in a similar manner.

Subplant curve incorporator 178 may modify the cost vector c to accountfor the resource consumption vectors in the decision matrix x. In someembodiments, subplant curve incorporator 178 removes (or sets to zero)any cost directly associated with the subplant loads (e.g., {dot over(Q)}_(Chiller,1 . . . n), and {dot over (Q)}_(Heater,1 . . . n), etc.)and adds economic costs associated with the resource consumptionrequired to produce the subplant loads. For example, for chillersubplant 16, subplant curve incorporator 178 may modify the cost vectorc as follows:

c=[ . . . 0_(n) . . . c _(elec,1 . . . n) c _(water,1 . . . n) . . .]^(T)

where 0_(n) is a n-dimensional zero vector indicating that the directeconomic cost of {dot over (Q)}_(Chiller,1 . . . n) is zero at each timestep, c_(elec,1 . . . n) is a n-dimensional vector indicating the perunit cost of electricity at each time step, and c_(water,1 . . . n) is an-dimensional vector indicating the per unit cost of water at each timestep. The modified cost vector associates an economic cost with theresources consumed to produce the subplant loads rather than thesubplant loads themselves. In some embodiments, the values forc_(elec,1 . . . n) and c_(water,1 . . . n) are utility rates obtainedfrom load/rate prediction circuit 122.

Subplant curve incorporator 178 may modify the inequality constraints toensure that the proper amount of each resource is consumed to serve thepredicted thermal energy loads. In some embodiments, subplant curveincorporator 178 formulates inequality constraints that force theresource usage for each resource in the epigraph of the correspondinglinearized subplant curve. For example, chiller subplant 16 may have alinearized subplant curve that indicates the electricity use of chillersubplant 16 (i.e., u_(Chiller,elec)) as a function of the cold waterproduction of chiller subplant 16 (i.e., u_(Chiller,elec)) Such alinearized subplant curve 700 is shown in FIG. 7 . The linearizedsubplant curve may include a first line segment connecting point [u₁,Q₁] to point [u₂, Q₂], a second line segment connecting point [u₂, Q₂]to point [u₃, Q₃], and a third line segment connecting point [u₃, Q₃] topoint [u₄, Q₄].

Subplant curve incorporator 178 may formulate an inequality constraintfor each piecewise segment of the subplant curve that constrains thevalue of u_(Chiller,elec) to be greater than or equal to the amount ofelectricity use defined by the line segment for the corresponding valueof {dot over (Q)}_(Chiller). The subplant curve constraints for theelectricity use of chiller subplant 16 can be placed in the form Ax≤b bydefining the A matrix and the b vector as follows:

${A = \begin{bmatrix}\ldots & {\left\lbrack {- \left( {u_{2} - u_{1}} \right)} \right\rbrack I_{n}} & \ldots & {\left\lbrack \left( {Q_{2} - Q_{1}} \right) \right\rbrack I_{n}} & 0_{n} & \ldots \\\ldots & {\left\lbrack {- \left( {u_{3} - u_{2}} \right)} \right\rbrack I_{n}} & \ldots & {\left\lbrack \left( {Q_{3} - Q_{2}} \right) \right\rbrack I_{n}} & 0_{n} & \ldots \\\ldots & {\left\lbrack {- \left( {u_{4} - u_{3}} \right)} \right\rbrack I_{n}} & \ldots & {\left\lbrack \left( {Q_{4} - Q_{3}} \right) \right\rbrack I_{n}} & 0_{n} & \ldots\end{bmatrix}},{b = \begin{bmatrix}{{Q_{1}u_{2}} - {Q_{2}u_{1}}} \\{{Q_{2}u_{3}} - {Q_{3}u_{2}}} \\{{Q_{3}u_{4}} - {Q_{4}u_{3}}}\end{bmatrix}}$

Similar inequality constraints can be formulated for other subplantcurves. For example, subplant curve incorporator 178 may generate a setof inequality constraints for the water consumptionu_(Chiller,water,1 . . . n) of chiller subplant 16 using the pointsdefining the linearized subplant curve for the water consumptionu_(Chiller,water,1 . . . n) of chiller subplant 16 as a function of coldwater production {dot over (Q)}_(Chiller). In some embodiments, thewater consumption of chiller subplant 16 is equal to the cold waterproduction and the linearized subplant curve for water consumptionincludes a single line segment connecting point [u₅, Q₅] to point [u₆,Q₆] (as shown in FIG. 5B). The subplant curve constraints for the coldwater consumption of chiller subplant 16 can be placed in the form Ax≤bby defining the A matrix and the b vector as follows:

A=[ . . . [−(u ₆ −u ₅)]I _(n) . . . 0_(n)[(Q ₆ −Q ₅)]I _(n) . . . ],b=[Q₅ u ₆ −Q ₆ u ₅]

Subplant curve incorporator 178 may repeat this process for eachsubplant curve for chiller subplant 16 and for the other subplants ofcentral plant 10 to define a set of inequality constraints for eachsubplant curve.

The inequality constraints generated by subplant curve incorporator 178ensure that high level optimization circuit 130 keeps the resourceconsumption above all of the line segments of the corresponding subplantcurve. In most situations, there is no reason for high leveloptimization circuit 130 to choose a resource consumption value thatlies above the corresponding subplant curve due to the economic costassociated with resource consumption. High level optimization circuit130 can therefore be expected to select resource consumption values thatlie on the corresponding subplant curve rather than above it.

The exception to this general rule is heat recovery chiller subplant 14.The equality constraints for heat recovery chiller subplant 14 providethat heat recovery chiller subplant 14 produces hot water at a rateequal to the subplant's cold water production plus the subplant'selectricity use. The inequality constraints generated by subplant curveincorporator 178 for heat recovery chiller subplant 14 allow high leveloptimization circuit 130 to overuse electricity to make more hot waterwithout increasing the amount of cold water production. This behavior isextremely inefficient and only becomes a realistic possibility when thedemand for hot water is high and cannot be met using more efficienttechniques. However, this is not how heat recovery chiller subplant 14actually operates.

To prevent high level optimization circuit 130 from overusingelectricity, subplant curve incorporator 178 may check whether thecalculated amount of electricity use (determined by the optimizationalgorithm) for heat recovery chiller subplant 14 is above thecorresponding subplant curve. In some embodiments, the check isperformed after each iteration of the optimization algorithm. If thecalculated amount of electricity use for heat recovery chiller subplant14 is above the subplant curve, subplant curve incorporator 178 maydetermine that high level optimization circuit 130 is overusingelectricity. In response to a determination that high level optimizationcircuit 130 is overusing electricity, subplant curve incorporator 178may constrain the production of heat recovery chiller subplant 14 at itscurrent value and constrain the electricity use of subplant 14 to thecorresponding value on the subplant curve. High level optimizationcircuit 130 may then rerun the optimization with the new equalityconstraints.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a ground loop circuit 152 and a heat exchanger circuit 154.In some embodiments, central plant 10 includes a heat exchangerconfigured to transfer heat between hot water loop 24 and condenserwater loop 28. In some embodiments, central plant 10 includes a groundloop that serves as heat rejection for chiller subplant 16 and/or heatextraction for heat recovery chiller subplant 14. Ground loop circuit152 and heat exchanger circuit 154 may be configured to modify theoptimization problem to account for heat transfer resulting fromoperation of the heat exchanger and/or the ground loop.

Ground loop circuit 152 may incorporate heat rejection to the groundloop into the optimization problem by changing the amount of electricityand water usage by chiller subplant 16. For example, for loadings up tothe heat rejection capacity of the ground loop, chiller subplant 16 mayuse an additional amount of electricity to run the ground loop pumps.The additional electricity usage may be constant or may vary per unit offlow through the ground loop. The amount of water production of chillersubplant 16 may be constant regardless of whether the ground loop isused.

Ground loop circuit 152 and heat exchanger circuit 154 may incorporateheat extraction from the ground loop and heat transfer between hot waterloop 24 and condenser water loop 28 into the optimization problem in asimilar manner. For example, ground loop circuit 152 and heat exchangercircuit 154 may use heat extraction from the ground loop and heattransfer between loops 24 and 28 to modify the load seen by the centralplant equipment. Ground loop circuit 152 may use the ground loop tocreate what appears as a false building load to the equipment, therebyallowing heat recovery chiller subplant 14 to operate as heat pumps whenthe building load does not add enough heat to the system. This outcomemay be optimal when the ratio between electricity prices and gas pricesis low such that it is less expensive to operate the ground loop and theheat exchanger using electricity than it would be to use natural gas togenerate heat in heater subplant 12.

Heat exchanger circuit 154 may use the heat exchanger to create whatappears to be a false hot water building load, thereby allowing heatrecovery chiller subplant 14 to operate as conventional chillers. Theexcess heat from heat recovery chiller subplant 14 may be transferredthrough the heat exchanger to condenser loop 28 and ultimately into theatmosphere or into the ground. In some embodiments, heat exchangercircuit 154 operates the heat exchanger to prevent condenser loop frombecoming overloaded. For example, heat exchanger circuit 154 may limitthe total heat rejected to the capacity of condenser loop 28 minus theheat produced by the conventional chillers.

Ground loop circuit 152 and heat exchanger circuit 154 may modify thedecision matrix x by adding a new decision vector for each type ofthermal energy load. The new decision vectors may represent theoverproduction of each thermal energy load for each time step k withinthe optimization period. For example, the modified decision matrix mayappear as follows:

x=[{dot over (Q)} _(Chiller,1 . . . n) ,{dot over (Q)}_(hrChiller,1 . . . n) ,{dot over (Q)} _(Heater,1 . . . n) ,{dot over(Q)} _(HotStorage,1 . . . n) ,{dot over (Q)} _(ColdStorage,1 . . . n) ,. . . . . . ,Q _(ColdUnmet,1 . . . n) ,Q _(HotUnmet,1 . . . n) {dot over(Q)} _(ColdOver,1 . . . n) {dot over (Q)} _(HotOver,1 . . . n)]^(T)

where {dot over (Q)}_(ColdOver,1 . . . n) and {dot over(Q)}_(HotOver,1 . . . n) are n-dimensional vectors representing theoverproduction rates of the cold thermal energy load and the hot thermalenergy load, respectively, for each time step k within the optimizationperiod.

Ground loop circuit 152 and heat exchanger circuit 154 may modify theequality constraints to account for any overproduced thermal energyloads. The overproduced thermal energy loads may be added to theequality constraints as slack variables that operate in the oppositedirection of the unmet loads. The modified equality constraints mayrequire that the predicted thermal energy loads plus any overproductionare equal to the total loads satisfied by subplants 12-22 plus anyunsatisfied thermal energy loads. The modified equality constraints canbe placed in the form Hx=g by defining the H matrix and the g vector asfollows:

$H = \begin{bmatrix}\left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & {\left\lbrack I_{h} \right\rbrack - \left\lbrack D_{- 1} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & {- \left\lbrack I_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack \\\left\lbrack 0_{h} \right\rbrack & \left( {1 + {u_{{elec},{hrChiller}}\left\lbrack I_{h} \right\rbrack}} \right. & \left\lbrack I_{h} \right\rbrack & \left\lbrack I_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack & {\left\lbrack I_{h} \right\rbrack - \left\lbrack D_{- 1} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & {- \left\lbrack I_{h} \right\rbrack}\end{bmatrix}$ ${\mathcal{g}} = \begin{bmatrix}{\hat{\ell}}_{{Cold},{1\ldots k}} \\{\hat{\ell}}_{{Hot},{1\ldots k}}\end{bmatrix}$

where [D⁻¹] is a lower diagonal matrix of ones. Ground loop circuit 152and heat exchanger circuit 154 may modify the cost vector c with theadditional cost of the pumping power per unit of overproduction requiredto run the ground loop and/or the heat exchanger.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a demand charge circuit 156. As discussed above, optimizationframework circuit 142 may formulate the optimization problem as:

${{\underset{x}{\arg\min}c^{T}x};{{{subject}{to}{Ax}} \leq b}},{{Hx} = {\mathcal{g}}}$

However, such a formulation does not account for the demand charge.

The demand charge is an additional charge imposed by some utilityproviders based on the maximum rate of energy consumption during anapplicable demand charge period. For example, the demand charge may beprovided in terms of dollars per unit of power (e.g., $/kW) and may bemultiplied by the peak power usage (e.g., kW) during a demand chargeperiod to calculate the demand charge. In some instances, the demandcharge can account for more than 15% of the electrical bill. Failure toinclude the demand charge in the optimization scheme can cause all ofthe equipment to turn on at the same time (e.g., the most efficient orlowest cost time). This would be optimal from a consumption coststandpoint. However, shifting some of the load in time may savethousands of dollars on demand while only costing a few dollars inconsumption cost.

Demand charge circuit 156 may be configured to modify the optimizationproblem to account for the demand charge. Incorporating the demandcharge into the optimization framework may greatly improve theperformance of the high level optimization. For example, including thedemand charge in the optimization framework may reduce the totaloperating costs of central plant 10 by an additional 5% on top of the8-10% cost reduction provided by other circuits of central plantcontroller 102. In various implementations, the savings provided bydemand charge circuit 156 and/or central plant controller 102 as a wholemay be greater than or less than the exemplary amounts defined hereindue to differences in plant configuration and/or energy costs.

Demand charge circuit 156 may account for the demand charge by modifyingthe cost function used by high level optimization circuit 130. Themodified cost function may be defined as:

${{\underset{x}{\arg\min}\left\lbrack {{c^{T}x} + {c_{demand}{\max\left( P_{{elec},k} \right)}}} \right\rbrack};{{{subject}{to}{Ax}} \leq b}},{{Hx} = {\mathcal{g}}}$

where c_(demand) is the demand charge (e.g., $/kW) for the applicabledemand charge period and P_(elec,k) is the total electrical powerconsumption of central plant 10 and the building/campus at time step k.The term max (P_(elec,k)) selects the peak electrical power consumptionat any time during the demand charge period. The demand chargec_(demand) and the demand charge period may be defined by the utilityrate information received from utilities 126 and may be provided to highlevel optimization circuit 130 by load/rate prediction circuit 122.

Incorporating the demand charge into the optimization frameworkcomplicates the optimization problem in two primary ways. First, thecost function is no longer linear due to the inclusion of the max( )function. Second, the consumption term c^(T)x calculates cost over aconsumption period defined by a time horizon, whereas the demand chargeterm c_(demand) max(P_(elec,k)) calculates cost over the demand chargeperiod. For example, the consumption period may be defined as the timeperiod beginning at the current time step k and ending at a future timestep k+h, where h represents the time horizon. The demand charge periodmay be defined by utilities 126 and provided to high level optimizationcircuit 130 along with the utility rate information. In some instances,the consumption period and the demand charge period may not be the same.This complicates the optimization problem by obfuscating potentialtrade-offs between control decisions that reduce the consumption term atthe expense of the demand charge term or vice versa.

Demand charge circuit 156 may modify the optimization problem toincorporate the demand charge term into the linear optimizationframework. For example, demand charge circuit 156 may modify thedecision matrix x by adding a new decision variable x_(peak) as follows:

x _(new)=[ . . . u _(Chiller,elec,1 . . . n) . . . u_(hpChiller,elec,1 . . . n) . . . u _(Heater,elec,1 . . . n) . . . x_(peak)]^(T)

where x_(peak) is the peak power consumption within the demand chargeperiod. Demand charge circuit 156 may modify the cost vector c asfollows:

c _(new)=[ . . . c _(elec,1 . . . n) . . . c _(elec,1 . . . n) . . . c_(elec,1 . . . n) . . . c _(demand)]^(T)

such that the demand charge c_(demand) is multiplied by the peak powerconsumption x_(peak)

Demand charge circuit 156 may formulate and/or apply inequalityconstraints to ensure that the peak power consumption x_(peak) isgreater than or equal to the maximum electric demand over the demandcharge period. I.e.:

x _(peak)≥max(u _(Chiller,elec,k) +u _(hpChiller,elec,k) +u_(Heater,elec,k) +P _(elec,campus,k))∀k∈horizon

This inequality constraint may be represented in the linear optimizationframework by defining the A matrix and the b vector as follows:

A=[ . . . [I _(h)] . . . [I _(h)] . . . −1],b=−P _(elec,campus,k)

During the high level optimization process, high level optimizationcircuit 130 may choose a x_(peak) that is equal to the maximumelectrical demand over the demand charge period to minimize the costassociated with x_(peak).

Demand charge circuit 156 may apply an inequality constraint to ensurethat the peak power consumption decision variable x_(peak) is greaterthan or equal to its previous value x_(peak,previous) during the demandcharge period. This inequality constraint may be represented in thelinear optimization framework by defining the A matrix and the b vectoras follows:

A=[−1],b=−x _(peak,previous)

Advantageously, the modifications to the decision variable matrix x, thecost vector c, and the inequality constraints provided by demand chargecircuit 156 allow the cost function to be written in a linear form asfollows:

${{{\underset{x}{\arg\min}\left\lbrack {c_{new}^{T}x_{new}} \right\rbrack} = {\arg\limits_{x}{\min\left\lbrack {{c^{T}x} + {c_{demand}x_{peak}}} \right\rbrack}}};{{{subject}{to}{Ax}} \leq b}},{{Hx} = {\mathcal{g}}}$

This linear form of the cost function can be used in the linearoptimization framework.

The cost function as written in the previous equation has componentsthat are over different time periods. For example, the consumption termc^(T)x is over the consumption period whereas the demand charge termc_(demand)x_(peak) is over the demand charge period. To properly makethe trade-off between increasing the demand charge versus increasing thecost of energy consumption, demand charge circuit 156 may apply aweighting factor to the demand charge term and/or the consumption term.For example, demand charge circuit 156 may divide the consumption termc^(T)x by the duration h of the consumption period (i.e., the timeperiod between the current time and the time horizon) and multiply bythe amount of time d_(demand) remaining in the current demand chargeperiod so that the entire cost function is over the demand chargeperiod. The new optimization function may be given by:

${{\underset{x}{\arg\min}\left\lbrack {{\frac{d_{demand}}{h}c^{T}x} + {c_{demand}x_{peak}}} \right\rbrack};{{{subject}{to}{Ax}} \leq b}},{{Hx} = {\mathcal{g}}}$whichisequivalentto:${{\underset{x}{\arg\min}\left\lbrack {{c^{T}x} + {\frac{h}{d_{demand}}c_{demand}x_{peak}}} \right\rbrack};{{{subject}{to}{Ax}} \leq b}},{{Hx} = {\mathcal{g}}}$

The latter form of the new optimization function has the advantage ofadjusting only one term of the function rather than several.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a load change penalty circuit 158. In some instances, highlevel optimization circuit 130 determines a solution to the optimizationproblem that includes significantly changing the load on one or more ofsubplants 12-22 within a relatively short timeframe. For example, thelowest cost solution from a resource consumption standpoint may involvetaking a subplant from off to full load and back to off again withinonly a few time steps. This behavior may result from high leveloptimization circuit 130 identifying small fluctuations in the economiccost of resources and operating central plant 10 accordingly to achievethe minimal economic cost. However, operating central plant 10 in such away may be undesirable due to various negative effects of rapidlychanging the subplant loads (e.g., increased equipment degradation),especially if the cost saved is relatively minimal (e.g., a few cents ordollars).

Load change penalty circuit 158 may modify the optimization problem tointroduce a penalty for rapidly changing the subplant loads. Forexample, load change penalty circuit 158 may modify the decision matrixx by adding a new decision vector for each subplant. The new decisionvectors represent the change in subplant load for each subplant from onetime step to the next. For example, load change penalty circuit 158 maymodify the decision matrix x as follows:

x=[ . . . {dot over (Q)} _(Chiller,1 . . . n) . . . {dot over (Q)}_(hrChiller,1 . . . n) . . . {dot over (Q)} _(Heater,1 . . . n) . . .δ_(Chiller1, . . . n)δ_(hrChiller1, . . . n)δ_(Heater,1 . . . n)]^(T)

where δ_(Chiller,1 . . . n), δ_(hrChiller,1 . . . n), andδ_(Heater,1 . . . n) are n-dimensional vectors representing the changein subplant load for {dot over (Q)}_(Chiller,1 . . . n), {dot over(Q)}_(hrChiller,1 . . . n), and {dot over (Q)}_(Heater,1 . . . n),respectively, at each time step k relative to the previous time stepk−1.

Load change penalty circuit 158 may modify the cost vector c to add acost associated with changing the subplant loads. For example, loadchange penalty circuit 158 may modify the cost vector c as follows:

C=[ . . . 0 _(n) . . . 0 _(n) . . . 0 _(n) . . . c _(δChiller,1 . . . n)c _(δhrChiller,1 . . . n) c _(δHeater,1 . . . n)]^(T)

Load change penalty circuit 158 may add constraints such that each ofthe load change variables δ cannot be less than the change in thecorresponding subplant load {dot over (Q)}. For example, the addedconstraints for chiller subplant 16 may have the following form:

${A = \begin{bmatrix}\ldots & {I_{h} - D_{- 1}} & \ldots & {- \left\lbrack I_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\ldots & {D_{- 1} - I_{h}} & \ldots & {- \left\lbrack I_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \end{bmatrix}},{b = \begin{bmatrix}\begin{bmatrix}{\overset{.}{Q}}_{{Chiller},{old}} \\0_{h - 1}\end{bmatrix} \\\begin{bmatrix}{- {\overset{.}{Q}}_{{Chiller},{old}}} \\0_{h - 1}\end{bmatrix} \\ \vdots \end{bmatrix}}$

where {dot over (Q)}_(Chiller,old) is the value for {dot over(Q)}_(chiller) at the previous time step. Similar constraints may beadded for each of subplants 12-22.

The constraints added by load change penalty circuit 158 require thatthe load change variables δ are greater than or equal to the magnitudeof the difference between the current value of the correspondingsubplant load {dot over (Q)} and the previous value of the subplant load{dot over (Q)}_(old). In operation, high level optimization circuit 130may select values for the load change variables δ that are equal to themagnitude of the difference due to the costs associated with the loadchange variables. In other words, high level optimization circuit 130may not choose to make the load change variables δ greater than theactual change in the corresponding subplant load because making the loadchange variables δ greater than necessary would be suboptimal.

Still referring to FIG. 4 , high level optimization circuit 130 is shownto include a tank forced full circuit 160. Tank forced full circuit 160may modify the optimization problem such that the thermal energy storage(TES) tanks are forced to full at the end of the optimization period.This feature provides increased robustness in the event of a subplantfailure and/or controller failure by ensuring that the TES tanks havesufficient stored thermal energy to satisfy building loads while thefailure is being repaired. For example, plant operators can use thestored thermal energy to meet the building loads while central plantcontroller 102 is brought back online.

Tank forced full circuit 160 may force the TES tanks to full byincreasing the cost of discharging the TES tanks. In some embodiments,tank forced full circuit 160 modifies the cost of discharging the TEStanks such that the discharge cost is higher than other costs in thecost function, but less than the cost of unmet loads. This forces highlevel optimization circuit 130 to take the benefit (i.e., negative cost)of charging the TES tanks to their maximum values.

Referring now to FIGS. 5A-B, two subplant curves 500 and 510 are shown,according to an exemplary embodiment. Subplant curves 500 and 510 may beused as subplant curves 140, as described with reference to FIG. 3 .Subplant curves 500 and 510 define the resource usage of a subplant(e.g., one of subplants 12-22) as a function of the subplant load. Eachsubplant curve may be specific to a particular subplant and a particulartype of resource used by the subplant. For example, subplant curve 500may define the electricity use 502 of chiller subplant 16 as a functionof the load 504 on chiller subplant 16, whereas subplant curve 510 maydefine the water use 506 of chiller subplant 16 as a function of theload 504 on chiller subplant 16. Each of subplants 12-22 may have one ormore subplant curves (e.g., one for each type of resource consumed bythe subplant).

In some embodiments, low level optimization circuit 132 generatessubplant curves 500 and 510 based on equipment models 120 (e.g., bycombining equipment models 120 for individual devices into an aggregatecurve for the subplant). Low level optimization circuit 132 may generatesubplant curves 500 and 510 by running the low level optimizationprocess for several different loads and weather conditions to generatemultiple data points. Low level optimization circuit 132 may fit a curveto the data points to generate the subplant curves. In otherembodiments, low level optimization circuit 132 provides the data pointsto high level optimization circuit 132 and high level optimizationcircuit 132 generates the subplant curves using the data points.

Referring now to FIG. 6 , another subplant curve 600 is shown, accordingto an exemplary embodiment. Subplant curve 600 defines the electricityuse of chiller subplant 16 (i.e., u_(Chiller,elec)) as a function of thecold water production of chiller subplant 16 (i.e., {dot over(Q)}_(Chiller)) In some embodiments, subplant curve 600 is generated bycombining efficiency curves for individual devices of chiller subplant16 (e.g., individual chillers, pumps, etc.). For example, each of thechillers in subplant 16 may have a device-specific efficiency curve thatdefines the amount of electricity use by the chiller as a function ofthe load on the chiller. Many devices operate less efficiently at higherloads and have device efficiency curves that are U-shaped functions ofload. Accordingly, combining multiple device efficiency curves to formsubplant curve 600 may result in subplant curve 600 having one or morewaves 602, as shown in FIG. 6 . Waves 602 may be caused by a singledevice loading up before it is more efficient to turn on another deviceto satisfy the subplant load.

Referring now to FIG. 7 , a linearized subplant curve 700 is shown,according to an exemplary embodiment. Subplant curve 700 defines theelectricity use of chiller subplant 16 (i.e., u_(Chiller,elec)) as afunction of the cold water production of chiller subplant 16 (i.e., {dotover (Q)}_(Chiller)) Subplant curve 700 may be generated by convertingsubplant curve 600 into a linearized convex curve. A convex curve is acurve for which a line connecting any two points on the curve is alwaysabove or along the curve (i.e., not below the curve). Convex curves maybe advantageous for use in the high level optimization because theyallow for an optimization process that is less computationally expensiverelative to an optimization process that uses non-convex functions.

In some embodiments, subplant curve 700 is generated by subplant curvelinearizer 176, as described with reference to FIG. 4 . Subplant curve700 may be created by generating a plurality of linear segments (i.e.,segments 702, 704, and 706) that approximate subplant curve 600 andcombining the linear segments into a piecewise-defined linearized convexcurve 700. Linearized subplant curve 700 is shown to include a firstlinear segment 702 connecting point [u₁, Q₁] to point [u₂, Q₂], a secondlinear segment 704 connecting point [u₂, Q₂] to point [u₃, Q₃], and athird linear segment 706 connecting point [u₃, Q₃] to point [u₄, Q₄].The endpoints of line segments 702-706 may be used to form constraintsthat force the electricity use of chiller subplant 16 in the epigraph ofthe linearized subplant curve 700.

Referring now to FIG. 8 , another subplant curve 800 is shown, accordingto an exemplary embodiment. Subplant curve 800 defines the energy use ofone of subplants 12-22 as a function of the load on the subplant forseveral different weather conditions. In some embodiments, subplantcurve 800 is generated by subplant curves circuit 170 using experimentaldata obtained from the low level optimization circuit 132. For example,subplant curve updater 172 may request resource usage data from lowlevel optimization circuit 132 for various combinations of loadconditions and environmental conditions. In the embodiment shown in FIG.8 , subplant curve updater 172 requests energy use data for eachcombination of temperature (e.g., 40° F., 50° F., 60° F., and 70° F.)and load (e.g., 170 tons, 330 tons, 500 tons, 830 tons, and 1000 tons).Low level optimization circuit 132 may perform the low leveloptimization process for the requested load and temperature combinationsand return an energy use value for each combination.

Subplant curve updater 172 may use the data points provided by low leveloptimization circuit 132 to find the best piecewise linear convexfunction that fits the data. For example, subplant curve updater 172 mayfit a first subplant curve 802 to the data points at 70° F., a secondsubplant curve 804 to the data points at 60° F., a third subplant curve806 to the data points at 50° F., and a fourth subplant curve 808 to thedata points at 40° F. Subplant curve updater 172 may store the generatedsubplant curves 802-808 in subplant curves database 174 for use in thehigh level optimization algorithm.

In some implementations, central plant controller 102 uses high leveloptimization circuit 130 as part of an operational tool to exercisereal-time control over central plant 10. In the operational tool, highlevel optimization circuit 130 may receive load and rate predictionsfrom load/rate prediction circuit 122 and subplant curves (or data thatcan be used to generate subplant curves) from low level optimizationcircuit 132. When implemented in the operational tool, high leveloptimization circuit 130 may determine optimal subplant loads for heatersubplant 12, heat recovery chiller subplant 14, chiller subplant 16, hotTES subplant 20, and/or cold TES subplant 22, as described withreference to FIGS. 2-4 . In some embodiments, high level optimizationcircuit 130 determines ground loop and heat exchanger transfer rates inaddition to the subplant loads. When implemented in the operationaltool, high level optimization circuit 130 may provide the determinedsubplant loads and/or transfer rates to low level optimization circuit132 for use in determining optimal on/off decisions and/or operatingsetpoints for the equipment of each subplant.

Referring now to FIG. 9 , a block diagram of a planning system 900 isshown, according to an exemplary embodiment. Planning system 900 may beconfigured to use optimization circuit 128 as part of a planning tool902 to simulate the operation of a central plant over a predeterminedtime period (e.g., a day, a month, a week, a year, etc.) for planning,budgeting, and/or design considerations. When implemented in planningtool 902, optimization circuit 128 may operate in a similar manner asdescribed with reference to FIGS. 2-4 . For example, optimizationcircuit 128 may use building loads and utility rates to determine anoptimal subplant load distribution to minimize cost over a simulationperiod. However, planning tool 902 may not be responsible for real-timecontrol of a building automation system or central plant.

In planning tool 902, high level optimization circuit 130 may receiveplanned loads and utility rates for the entire simulation period. Theplanned loads and utility rates may be defined by input received from auser via a client device 922 (e.g., user-defined, user selected, etc.)and/or retrieved from a plan information database 926. High leveloptimization circuit 130 uses the planned loads and utility rates inconjunction with subplant curves from low level optimization circuit 132to determine optimal subplant loads (i.e., an optimal dispatch schedule)for a portion of the simulation period.

The portion of the simulation period over which high level optimizationcircuit 130 optimizes the subplant loads may be defined by a predictionwindow ending at a time horizon. With each iteration of theoptimization, the prediction window is shifted forward and the portionof the dispatch schedule no longer in the prediction window is accepted(e.g., stored or output as results of the simulation). Load and ratepredictions may be predefined for the entire simulation and may not besubject to adjustments in each iteration. However, shifting theprediction window forward in time may introduce additional planinformation (e.g., planned loads and/or utility rates) for thenewly-added time slice at the end of the prediction window. The new planinformation may not have a significant effect on the optimal dispatchschedule since only a small portion of the prediction window changeswith each iteration.

In some embodiments, high level optimization circuit 130 requests all ofthe subplant curves used in the simulation from low level optimizationcircuit 132 at the beginning of the simulation. Since the planned loadsand environmental conditions are known for the entire simulation period,high level optimization circuit 130 may retrieve all of the relevantsubplant curves at the beginning of the simulation. In some embodiments,low level optimization circuit 132 generates functions that map subplantproduction to equipment level production and resource use when thesubplant curves are provided to high level optimization circuit 130.These subplant to equipment functions may be used to calculate theindividual equipment production and resource use (e.g., in apost-processing circuit) based on the results of the simulation.

Still referring to FIG. 9 , planning tool 902 is shown to include acommunications interface 904 and a processing circuit 906.Communications interface 904 may include wired or wireless interfaces(e.g., jacks, antennas, transmitters, receivers, transceivers, wireterminals, etc.) for conducting data communications with varioussystems, devices, or networks. For example, communications interface 904may include an Ethernet card and port for sending and receiving data viaan Ethernet-based communications network and/or a WiFi transceiver forcommunicating via a wireless communications network. Communicationsinterface 904 may be configured to communicate via local area networksor wide area networks (e.g., the Internet, a building WAN, etc.) and mayuse a variety of communications protocols (e.g., BACnet, IP, LON, etc.).

Communications interface 904 may be a network interface configured tofacilitate electronic data communications between planning tool 902 andvarious external systems or devices (e.g., client device 922, resultsdatabase 928, plan information database 926, etc.). For example,planning tool 902 may receive planned loads and utility rates fromclient device 922 and/or plan information database 926 viacommunications interface 904. Planning tool 902 may use communicationsinterface 904 to output results of the simulation to client device 922and/or to store the results in results database 928.

Still referring to FIG. 9 , processing circuit 906 is shown to include aprocessor 910 and memory 912. Processor 910 may be a general purpose orspecific purpose processor, an application specific integrated circuit(ASIC), one or more field programmable gate arrays (FPGAs), a group ofprocessing components, or other suitable processing components.Processor 910 may be configured to execute computer code or instructionsstored in memory 912 or received from other computer readable media(e.g., CDROM, network storage, a remote server, etc.).

Memory 912 may include one or more devices (e.g., memory units, memorydevices, storage devices, etc.) for storing data and/or computer codefor completing and/or facilitating the various processes described inthe present disclosure. Memory 912 may include random access memory(RAM), read-only memory (ROM), hard drive storage, temporary storage,non-volatile memory, flash memory, optical memory, or any other suitablememory for storing software objects and/or computer instructions. Memory912 may include database components, object code components, scriptcomponents, or any other type of information structure for supportingthe various activities and information structures described in thepresent disclosure. Memory 912 may be communicably connected toprocessor 910 via processing circuit 906 and may include computer codefor executing (e.g., by processor 906) one or more processes describedherein.

Still referring to FIG. 9 , memory 912 is shown to include a GUI engine926, web services 914, and configuration tools 918. In an exemplaryembodiment, GUI engine 916 includes a graphical user interface componentconfigured to provide graphical user interfaces to a user for selectingor defining plan information for the simulation (e.g., planned loads,utility rates, environmental conditions, etc.). Web services 914 mayallow a user to interact with planning tool 902 via a web portal and/orfrom a remote system or device (e.g., an enterprise controlapplication).

Configuration tools 918 can allow a user to define (e.g., via graphicaluser interfaces, via prompt-driven “wizards,” etc.) various parametersof the simulation such as the number and type of subplants, the deviceswithin each subplant, the subplant curves, device-specific efficiencycurves, the duration of the simulation, the duration of the predictionwindow, the duration of each time step, and/or various other types ofplan information related to the simulation. Configuration tools 918 canpresent user interfaces for building the simulation. The user interfacesmay allow users to define simulation parameters graphically. In someembodiments, the user interfaces allow a user to select a pre-stored orpre-constructed simulated plant and/or plan information (e.g., from planinformation database 926) and adapt it or enable it for use in thesimulation.

Still referring to FIG. 9 , memory 912 is shown to include optimizationcircuit 128. Optimization circuit 128 may use the planned loads andutility rates to determine optimal subplant loads over a predictionwindow. The operation of optimization circuit 128 may be the same orsimilar as previously described with reference to FIGS. 2-4 . With eachiteration of the optimization process, optimization circuit 128 mayshift the prediction window forward and apply the optimal subplant loadsfor the portion of the simulation period no longer in the predictionwindow. Optimization circuit 128 may use the new plan information at theend of the prediction window to perform the next iteration of theoptimization process. Optimization circuit 128 may output the appliedsubplant loads to reporting applications 930 for presentation to aclient device 922 (e.g., via user interface 924) or storage in resultsdatabase 928.

Still referring to FIG. 9 , memory 912 is shown to include reportingapplications 930. Reporting applications 930 may receive the optimizedsubplant loads from optimization circuit 128 and, in some embodiments,costs associated with the optimized subplant loads. Reportingapplications 930 may include a web-based reporting application withseveral graphical user interface (GUI) elements (e.g., widgets,dashboard controls, windows, etc.) for displaying key performanceindicators (KPI) or other information to users of a GUI. In addition,the GUI elements may summarize relative energy use and intensity acrossvarious plants, subplants, or the like. Other GUI elements or reportsmay be generated and shown based on available data that allow users toassess the results of the simulation. The user interface or report (orunderlying data engine) may be configured to aggregate and categorizesubplant loads and the costs associated therewith and provide theresults to a user via a GUI. The GUI elements may include charts orhistograms that allow the user to visually analyze the results of thesimulation. An exemplary output that may be generated by reportingapplications 930 is shown in FIG. 10 .

Referring now to FIG. 10 , several graphs 1000 illustrating theoperation of planning tool 902 are shown, according to an exemplaryembodiment. With each iteration of the optimization process, planningtool 902 selects an optimization period (i.e., a portion of thesimulation period) over which the optimization is performed. Forexample, planning tool 902 may select optimization period 1002 for usein the first iteration. Once the optimal load distribution 1010 has beendetermined, planning tool 902 may select a portion 1018 of loaddistribution 1010 to send to plant dispatch 1030. Portion 1018 may bethe first b time steps of load distribution 1010. Planning tool 902 mayshift the optimization period 1002 forward in time, resulting inoptimization period 1004. The amount by which the prediction window isshifted may correspond to the duration of time steps b.

Planning tool 902 may repeat the optimization process for optimizationperiod 1004 to determine the optimal plant load distribution 1012.Planning tool 902 may select a portion 1020 of plant load distribution1012 to send to plant dispatch 1030. Portion 1020 may be the first btime steps of load distribution 1012. Planning tool 902 may then shiftthe prediction window forward in time, resulting in optimization period1006. This process may be repeated for each subsequent optimizationperiod (e.g., optimization periods 1006, 1008, etc.) to generate updatedload distributions (e.g., load distributions 1014, 1016, etc.) and toselect portions of each load distribution (e.g., portions 1022, 1024) tosend to plant dispatch 1030. Plant dispatch 1030 includes the first btime steps 1018-1024 from each of optimization periods 1002-1008. Oncethe optimal subplant load distribution 1030 is compiled for the entiresimulation period, the results may be sent to reporting applications930, results database 928, and/or client device 922, as described withreference to FIG. 9 .

Referring now to FIG. 11 , a flowchart of a process 1100 for optimizingcost in a central plant is shown, according to an exemplary embodiment.In various implementations, process 1100 may be performed by centralplant controller 102 or planning tool 902. The central plant may includea plurality of subplants (e.g., subplants 12-22) configured to serve theenergy loads of a building or campus. The central plant may be an actualplant (e.g., central plant 10) or a simulated central plant including aplurality of simulated subplants.

Process 1100 is shown to include receiving load prediction data andutility rate data (step 1102). The load prediction data may includepredicted or planned thermal energy loads for a building or campus foreach time step k (e.g., k=1 n) of an optimization period. The loadprediction data may include predicted or planned values one or moredifferent types of loads for the building or campus. For example, theload prediction data may include a predicted hot water load

_(Hot,k) and a predicted cold water load

_(Cold,k) for each time step k within the prediction window.

In some embodiments, the load prediction data are based on weatherforecasts from a weather service and/or feedback from the building orcampus. Feedback from the building or campus may include various typesof sensory inputs (e.g., temperature, flow, humidity, enthalpy, etc.) orother data relating to the controlled building (e.g., inputs from a HVACsystem, a lighting control system, a security system, a water system,etc.). The load predictions may also be based on predictions of heatgenerated by data center equipment, for example as described herein withreference to data center heat prediction circuit 300. In someembodiments, the load prediction data are generated by load/rateprediction circuit 122, as described with reference to FIG. 2 . Forexample, the load prediction data may be based on a measured electricload and/or previous measured load data from the building or campus. Theload prediction data may be a function of a given weather forecast({circumflex over (ϕ)}_(w)), a day type (day), the time of day (t),and/or previous measured load data (Y_(k−1)). Such a relationship isexpressed in the following equation:

_(k) =f({circumflex over (ϕ)}_(w) ,day,t|Y _(k−1))

The utility rate data may indicate a cost or price per unit of one ormore resources (e.g., electricity, natural gas, water, etc.) consumed bythe central plant to serve the thermal energy loads of the building orcampus at each time step k in the prediction window. In someembodiments, the utility rates are time-variable rates. For example, theprice of electricity may be higher at certain times of day or days ofthe week (e.g., during high demand periods) and lower at other times ofday or days of the week (e.g., during low demand periods). The utilityrates may define various time periods and a cost per unit of a resourceduring each time period. Utility rates may be actual rates (e.g.,received from utilities 126) or predicted utility rates (e.g., estimatedby load/rate prediction circuit 122).

In some embodiments, the utility rates include demand charges for one ormore of the resources consumed by the central plant. A demand charge maydefine a separate cost based on the maximum usage of a particularresource (e.g., maximum energy consumption) during a demand chargeperiod. The utility rates may define various demand charge periods andone or more demand charges associated with each demand charge period. Insome instances, demand charge periods may overlap partially orcompletely with each other and/or with the prediction window. Theutility rate data may include time-variable (e.g., hourly) prices, amaximum service level (e.g., a maximum rate of consumption allowed bythe physical infrastructure or by contract) and, in the case ofelectricity, a demand charge or a charge for the peak rate ofconsumption within a certain period.

Still referring to FIG. 11 , process 1100 is shown to include generatingan objective function that expresses a total monetary cost of operatingthe central plant as a function of the utility rate data and an amountof resources consumed by the central plant (step 1104). In someembodiments, the objective function is a high level cost function J_(HL)for the central plant. The high level cost function J_(HL) may representthe sum of the monetary costs of each utility consumed by the centralplant for the duration of the optimization period. For example, the highlevel cost function J_(HL) may be described using the followingequation:

${J_{HL}\left( \theta_{HL} \right)} = {\sum\limits_{k = 1}^{n_{h}}{\sum\limits_{i = 1}^{n_{s}}\left\lbrack {\sum\limits_{j = 1}^{n_{u}}{{t_{s} \cdot c_{jk}}{u_{jik}\left( \theta_{HL} \right)}}} \right\rbrack}}$

where n_(h) is the number of time steps k in the optimization period,n_(s) is the number of subplants, t_(s) is the duration of a time step,c_(jk) is the economic cost of utility j at a time step k of theoptimization period, and u_(jik) is the rate of use of utility j bysubplant i at time step k.

In some embodiments, the objective function is generated using a linearprogramming framework. For example, step 1104 may include generating anobjective function of the form:

${{\underset{x}{argmin}c^{T}x};{{{subject}{to}{}{Ax}} \leq b}},{{Hx} = g}$

where c is a cost vector, x is a decision matrix, A and b are a matrixand vector (respectively) which describe inequality constraints on thevariables in the decision matrix x, and H and g are a matrix and vector(respectively) which describe equality constraints on the variables inthe decision matrix x. In other embodiments, the objective function maybe generated using any of a variety of other optimization frameworks(e.g., quadratic programming, linear-fractional programming, nonlinearprogramming, combinatorial algorithms, etc.).

In some embodiments, step 1104 includes formulating the decision matrixx. The loads across each of subplants 12-22 may be the decisionvariables in the decision matrix x. For example, for a central plantthat includes chillers, heat recovery chillers, hot water generators,and thermal energy storage, step 1104 may include formulating thedecision matrix x as:

x=[{dot over (Q)} _(Chiller,1 . . . n,{dot over (Q)})_(hrChiller,1 . . . n) ,{dot over (Q)} _(Heater,1 . . . n) ,{dot over(Q)} _(HotStorage,1 . . . n) ,{dot over (Q)}_(ColdStorage,1 . . . n)]^(T)

where {dot over (Q)}_(Chiller,1 . . . n), {dot over(Q)}_(hrChiller,1 . . . n), {dot over (Q)}_(Heater,1 . . . n), {dot over(Q)}_(HotStorage,1 . . . n), and {dot over (Q)}_(ColdStorage,1 . . . n)are n-dimensional vectors representing the thermal energy load assignedto chiller subplant 16, heat recovery chiller subplant 14, heatersubplant 12, hot TES subplant 20, and cold TES subplant 22,respectively, for each of the n time steps within the optimizationperiod.

In some embodiments, step 1104 includes generating the decision matrix xto include one or more decision vectors representing the resourceconsumption of each subplant. For example, for a central plant thatincludes a chiller subplant, step 1104 may include generating thedecision matrix x as follows:

x=[ . . . Q _(Chiller,1 . . . n) . . . u _(Chiller,elec,1 . . . n) u_(Chiller,water,1 . . . n) . . . ]T

where u_(Chiller,elec,1 . . . n) and u_(Chiller,water,1 . . . n) aren-dimensional vectors representing the amount of electrical consumptionand water consumption, respectively, by the chiller subplant at eachtime step k.

Step 1104 may include adding one or more resource consumption vectors tomatrix x for each of subplants 12-22. The decision vectors added in step1104 for a given subplant may represent an amount of resourceconsumption for each resource consumed by the subplant (e.g., water,electricity, natural gas, etc.) at each time step k within theoptimization period. For example, if a heater subplant consumes naturalgas, electricity, and water, step 1104 may include adding a decisionvector u_(Heater,gas,1 . . . n) representing an amount of natural gasconsumed by the heater subplant at each time step, a decision vectoru_(Heater,elec,1 . . . n) representing an amount of electricity consumedby the heater subplant at each time step, and a decision vectoru_(Heater,water,1 . . . n) representing an amount of water consumed bythe heater subplant at each time step. Step 1104 may include addingresource consumption vectors for other subplants in a similar manner.

In some embodiments, step 1104 includes generating the cost vector c.Generating the cost vector c may include adding economic costsassociated with the resource consumption required to produce thesubplant loads. For example, the decision matrix x provided above, step1104 may include generating the cost vector c as follows:

c=[ . . . 0 _(n) . . . c _(elec,1 . . . n) c _(water,1 . . . n) . . .]^(T)

where 0_(n) is a n-dimensional zero vector indicating that the directeconomic cost of {dot over (Q)}_(Chiller,1 . . . n) is zero at each timestep, c_(elec,1 . . . n) is a n-dimensional vector indicating the perunit cost of electricity at each time step, and c_(water,1 . . . n) is an-dimensional vector indicating the per unit cost of water at each timestep. The cost vector associates an economic cost with the resourcesconsumed to produce the subplant loads rather than the subplant loadsthemselves. In some embodiments, the values for c_(elec,1 . . . n) andc_(water,1 . . . n) are utility rates obtained from the utility ratedata received in step 1102.

In some embodiments, step 1104 includes generating the A matrix and theb vector which describe the inequality constraints, and the H matrix andthe g vector which describe the equality constraints. The inequalityconstraints and equality constraints may be generated by inequalityconstraints circuit 146 and equality constraints circuit 148, asdescribed with reference to FIG. 4 . For example, step 1104 may includegenerating inequality constraints that constrain the decision variablesin matrix x to be less than or equal to maximum capacities for thecorresponding central plant equipment and less than or equal to maximumcharge/discharge rates for thermal energy storage. Step 1104 may includegenerating inequality constraints that prevent charging the thermalenergy storage above maximum capacity and/or discharging the thermalenergy storage below zero. Step 1104 may include generating equalityconstraints that ensure the building energy loads are satisfied at eachof the time steps in the prediction window.

In some embodiments, step 1104 includes modifying the objective functionto account for unmet loads (e.g., as described with reference to unmetloads circuit 150), to account for heat extraction or rejection to aground loop (e.g., as described with reference to ground loop circuit152), to account for heat exchange between the hot water loop and thecondenser water loop (e.g., as described with reference to heatexchanger circuit 154), to account for subplant curves that are notsimple linear functions of load (e.g., as described with reference tosubplant curves circuit 170), and/or to force the thermal energy storagetanks to full at the end of the prediction window (e.g., as describedwith reference to tank forced full circuit 160). Modifying the objectivefunction may include modifying the decision matrix x, the cost vector c,the A matrix and the b vector which describe the inequality constraints,and/or the H matrix and the g vector which describe the equalityconstraints.

Still referring to FIG. 11 , process 1100 is shown to include modifyingthe objective function to account for a demand charge (step 1106). Step1106 is an optional step that may be performed by demand charge circuit156 to account for a demand charge that may be imposed by utilityproviders in some pricing scenarios. The demand charge is an additionalcharge imposed by some utility providers based on the maximum rate ofenergy consumption during an applicable demand charge period. Forexample, the demand charge may be provided in terms of dollars per unitof power (e.g., $/kW) and may be multiplied by the peak power usage(e.g., kW) during a demand charge period to calculate the demand charge.

Accounting for the demand charge may include modifying the variouscomponents of the objective function such as the decision matrix x, thecost vector c, and/or the A matrix and the b vector which describe theinequality constraints. The modified objective function may be definedas:

${{\underset{x}{argmin}\left\lbrack {{c^{T}x} + {c_{demand}{\max\left( P_{{elec},k} \right)}}} \right\rbrack};{{{subject}{to}{}{Ax}} \leq b}},{{Hx} = g}$

where c_(demand) is the demand charge for the applicable demand chargeperiod and P_(elec,k) is the total electrical power consumption of thecentral plant and the building/campus at time step k. The termmax(P_(elec,k)) selects the peak electrical power consumption at anytime during the demand charge period. The demand charge c_(demand) andthe demand charge period may be defined by the utility rate informationreceived in step 1102.

Step 1106 may include modifying the decision matrix x by adding a newdecision variable x_(peak) as follows:

x _(new=)[ . . . u _(Chiller,elec,1 . . . n) . . . u_(hpChiller,elec,1 . . . n) . . . u _(Heater,elec,1 . . . n) . . . x_(peak)]^(T)

where x_(peak) is the peak power consumption within the optimizationperiod. Step 1106 may include modifying the cost vector c as follows:

c _(new)=[ . . . c _(elec,1 . . . n) . . . c _(elec,1 . . . n) . . . c_(elec,1 . . . n) . . . c _(demand)]^(T)

such that the demand charge c_(demand) is multiplied by the peak powerconsumption x_(peak).

Step 1106 may include generating and/or imposing inequality constraintsto ensure that the peak power consumption x_(peak) is greater than orequal to the maximum electric demand for each time step in theoptimization period. I.e.:

x _(peak)≥max(u _(Chiller,elec,k) +u _(hpChiller,elec,k) +u_(Heater,elec,k) +P _(elec,campus,k))∀k∈horizon

This inequality constraint may be represented in the linear optimizationframework by defining the A matrix and the b vector as follows:

A=[ . . . [I _(h)] . . . [I _(h)] . . . [I _(h)] . . . −1],b=−P_(elec,campus,k)

Step 1106 may include generating and/or imposing an inequalityconstraint to ensure that the peak power consumption decision variablex_(peak) is greater than or equal to its previous valuex_(peak,previous) during the demand charge period. This inequalityconstraint may be represented in the linear optimization framework bydefining the A matrix and the b vector as follows:

A=[ . . . −1],b=−x _(peak,previous)

Advantageously, the modifications to the decision variable matrix x, thecost vector c, and the inequality constraints in step 1106 may allow theobjective function to be written in a linear form as follows:

${{{\underset{x}{argmin}\left\lbrack {c_{new}^{T}x_{new}} \right\rbrack} = {\underset{x}{argmin}\left\lbrack {{c^{T}x} + {c_{demand}x_{peak}}} \right\rbrack}};{{{subject}{to}{Ax}} \leq b}},{{Hx} = g}$

This linear form of the objective function can be used in the linearoptimization framework.

In some embodiments, step 1106 includes applying a weighting factor toat least one of the consumption term and the demand charge term of theobjective function. For example, the objective function as written inthe previous equation has components that are over different timeperiods. The consumption term c^(T)x is over the consumption periodwhereas the demand charge term c_(demand)x_(peak) is over the demandcharge period. To properly make the trade-off between increasing thedemand charge versus increasing the cost of energy consumption, step1106 may include applying a weighting factor to the demand charge termand/or the consumption term. For example, step 1106 may include dividingthe consumption term c^(T)x by the duration h of the consumption period(i.e., the time period between the current time and the time horizon)and multiplying by the amount of time d_(demand) remaining in thecurrent demand charge period so that the entire objective function isover the demand charge period. The new optimization function may begiven by:

${{\underset{x}{argmin}\left\lbrack {{\frac{d_{demand}}{h}c^{T}x} + {c_{demand}x_{peak}}} \right\rbrack};{{{subject}{to}{Ax}} \leq b}},{{Hx} = g}$

which is equivalent to:

${{\underset{x}{argmin}\left\lbrack {{c^{T}x} + {\frac{h}{d_{demand}}c_{demand}x_{peak}}} \right\rbrack};{{{subject}{to}{Ax}} \leq b}},{{Hx} = g}$

The latter form of the new optimization function has the advantage ofadjusting only one term of the function rather than several.

Still referring to FIG. 11 , process 1100 is shown to include modifyingthe objective function to account for a load change penalty (step 1108).Step 1108 is an optional step that may be performed by load changepenalty circuit 158 to account for the cost of changing the loadsassigned to each of the subplants. In some instances, the lowest costsolution from a resource consumption standpoint may involve taking asubplant from off to full load and back to off again within only a fewtime steps. However, operating the central plant in such a way may beundesirable due to various negative effects of rapidly changing thesubplant loads (e.g., increased equipment degradation), especially ifthe cost saved is relatively minimal (e.g., a few cents or dollars).

Step 1108 may include modifying the objective function to introduce apenalty for rapidly changing the subplant loads. In some embodiments,step 1108 includes modifying the decision matrix x by adding a newdecision vector for each subplant. The new decision vectors representthe change in subplant load for each subplant from one time step to thenext. For example, step 1108 may include modifying the decision matrix xas follows:

x=[ . . . {dot over (Q)} _(Chiller,1 . . . n) . . . {dot over (Q)}_(hrChiller,1 . . . n) . . . {dot over (Q)} _(Heater,1 . . . n) . . .δ_(Chiller,1 . . . n)δ_(hrChiller,1 . . . n)δ_(Heater,1 . . . n)]^(T)

where δ_(Chiller,1 . . . n), δ_(hrChiller,1 . . . n), andδ_(Heater,1 . . . n) are n-dimensional vectors representing the changein subplant load for {dot over (Q)}_(Chiller,1 . . . n), {dot over(Q)}_(hrChiller,1 . . . n), and {dot over (Q)}_(Heater,1 . . . n),respectively, at each time step k relative to the previous time stepk−1.

Step 1108 may include modifying the cost vector c to add a costassociated with changing the subplant loads. For example, step 1108 mayinclude modifying the cost vector c as follows:

c=[ . . . 0_(n) . . . 0_(n) . . . 0_(n) c _(δChiller,1 . . . n) c_(δhrChiller,1 . . . n) c _(δHeater,1 . . . n)]^(T)

Step 1108 may include adding constraints such that each of the loadchange variables δ cannot be less than the change in the correspondingsubplant load {dot over (Q)}. For example, the added constraints for achiller subplant may have the following form:

${A = \begin{bmatrix}\ldots & {I_{h} - D_{- 1}} & \ldots & {- \left\lbrack I_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\\ldots & {D_{- 1} - I_{h}} & \ldots & {- \left\lbrack I_{h} \right\rbrack} & \left\lbrack 0_{h} \right\rbrack & \left\lbrack 0_{h} \right\rbrack \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \end{bmatrix}},{b = \begin{bmatrix}\begin{bmatrix}{\overset{.}{Q}}_{{Chiller},{old}} \\0_{h - 1}\end{bmatrix} \\\begin{bmatrix}{- {\overset{.}{Q}}_{{Chiller},{old}}} \\0_{h - 1}\end{bmatrix} \\ \vdots \end{bmatrix}}$

where {dot over (Q)}_(Chiller,old) is the value for {dot over(Q)}_(Chiller) at the previous time step. Similar constraints may beadded for each of subplants 12-22. The constraints added by in step 1108may require that the load change variables δ are greater than or equalto the magnitude of the difference between the current value of thecorresponding subplant load {dot over (Q)} and the previous value of thesubplant load {dot over (Q)}_(old).

Still referring to FIG. 11 , process 1100 is shown to include optimizingthe objective function over an optimization period subject to a set ofconstraints to determine an optimal distribution of energy loads overmultiple groups of central plant equipment (step 1110). The set ofconstraints may include the inequality constraints and the equalityconstraints formulated in steps 1104, 1106, and/or 1108. Optimizing theobjective function may include determining an optimal decision matrix x*that minimizes the cost function c^(T)x. The optimal decision matrix x*may correspond to the optimal decisions θ*_(HL) (for each time step kwithin an optimization period) that minimize the high level costfunction J_(HL), as described with reference to FIG. 3 .

Step 1110 may include using any of a variety of linear optimizationtechniques to determine the optimal decision matrix. For example, step1110 may include using basis exchange algorithms (e.g., simplex,crisscross, etc.), interior point algorithms (e.g., ellipsoid,projective, path-following, etc.), covering and packing algorithms,integer programming algorithms (e.g., cutting-plant, branch and bound,branch and cut, branch and price, etc.), or any other type of linearoptimization algorithm or technique to solve for the optimal decisionmatrix subject to the optimization constraints. For embodiments in whichnonlinear optimization is used, step 1110 may include using any of avariety of nonlinear optimization techniques to solve for the optimaldecision matrix. The result of step 1110 may be an optimal distributionof energy loads over the multiple groups of subplant equipment (i.e.,the multiple subplants) for each of the time steps k.

Still referring to FIG. 11 , process 1100 is shown to include using theoptimal distribution of energy loads to determine optimal operatingstatuses for individual devices of the central plant equipment (step1112). In some embodiments, step 1112 is performed by low leveloptimization circuit 132, as described with reference to FIGS. 2-4 . Forexample, step 1112 may include using the subplant loads determined instep 1110 to determine optimal low level decisions θ*_(LL) (e.g. binaryon/off decisions, flow setpoints, temperature setpoints, etc.) for thecentral plant equipment. In some embodiments, step 1112 is performed foreach of the plurality of subplants.

Step 1112 may include determining which devices of each subplant to useand/or the operating setpoints for such devices that will achieve thesubplant load setpoint while minimizing energy consumption. The lowlevel optimization performed in step 1112 may be described using thefollowing equation:

$\theta_{LL}^{*} = {\arg\min\limits_{\theta_{LL}}{J_{LL}\left( \theta_{LL} \right)}}$

where θ*_(LL) contains the optimal low level decisions and J_(LL) is thelow level cost function.

To find the optimal low level decisions θ*_(LL), step 1112 may includeminimizing the low level cost function J_(LL). The low level costfunction J_(LL) may represent the total energy consumption for all ofthe equipment in the applicable subplant. The low level cost functionJ_(LL) may be described using the following equation:

${J_{LL}\left( \theta_{LL} \right)} = {\sum\limits_{j = 1}^{N}{t_{s} \cdot b_{j} \cdot {u_{j}\left( \theta_{LL} \right)}}}$

where N is the number of devices of in the subplant, t_(s) is theduration of a time step, b_(j) is a binary on/off decision (e.g., 0=off,1=on), and u_(j) is the energy used by device j as a function of thesetpoint θ_(LL). Each device may have continuous variables which can bechanged to determine the lowest possible energy consumption for theoverall input conditions.

In some embodiments, step 1112 includes minimizing the low level costfunction J_(LL) subject to inequality constraints based on thecapacities of the subplant equipment and equality constraints based onenergy and mass balances. In some embodiments, the optimal low leveldecisions θ*_(LL) are constrained by switching constraints defining ashort horizon for maintaining a device in an on or off state after abinary on/off switch. The switching constraints may prevent devices frombeing rapidly cycled on and off.

Step 1112 may include determining optimum operating statuses (e.g., onor off) for a plurality of devices of the central plant equipment.According to an exemplary embodiment, the on/off combinations may bedetermined using binary optimization and quadratic compensation. Binaryoptimization may minimize a cost function representing the powerconsumption of devices in the applicable subplant. In some embodiments,non-exhaustive (i.e., not all potential combinations of devices areconsidered) binary optimization is used. Quadratic compensation may beused in considering devices whose power consumption is quadratic (andnot linear).

Step 1112 may include determining optimum operating setpoints forequipment using nonlinear optimization. Nonlinear optimization mayidentify operating setpoints that further minimize the low level costfunction J_(LL). In some embodiments, step 1112 includes providing theon/off decisions and setpoints to building automation system 108 for usein controlling the central plant equipment 60.

HVAC Control for Data Center

Referring now to FIG. 12 , a block diagram of an HVAC system 1299 thatserves a data center 1200 is shown, according to an exemplaryembodiment. FIG. 12 illustrates the variety of heat transfers thataffect the indoor air temperature T_(ia) of the indoor air 1201 in datacenter 1200 of building 10. Data center 1200 is a room, floor, area,etc. of a building that houses data center equipment 1202. In general,the primary goal of the HVAC system 1299 is to maintain the indoor airtemperature T_(ia) in the data center 1200 at or around a desiredtemperature to maximizes the efficiency of the data center equipment1202 in the data center 1200 or to meet other needs of the data center1200. The HVAC system is a central plant as described with reference toFIGS. 1-11 or another type of HVAC system (e.g., an HVAC system providedfor the data center 1200).

As shown in FIG. 12 , the indoor air temperature T_(ia) of the datacenter 1200 has a thermal capacitance C_(ia). The indoor air temperatureTia is affected by a variety of heat transfers {dot over (Q)} into thedata center 1200, as described in detail below. It should be understoodthat although all heat transfers {dot over (Q)} (e.g., thermal energy)are shown in FIG. 12 as directed into the data center 1200, the value ofone or more of the heat transfers {dot over (Q)} may be negative, suchthat heat flows out of the data center 1200.

The data center equipment 1202 contributes data center equipment heattransfer {dot over (Q)}_(DCE) to the data center 1200. The data centerequipment 1202 includes servers, processor, computers, and otherelectronic devices in the data center 1200 that generate heat throughelectrical resistance during operation of the data center equipment1202. The heat transferred into the data center 1200 by the data centerequipment 1202 is denoted as {dot over (Q)}_(DCE). The heat transfer{dot over (Q)}_(DCE) may be predicted/modelled by the data equipmentheat predictor circuit 300 as described below.

The building mass 1204 contributes building mass heat transfer {dot over(Q)}_(m) to the data center 1200. The building mass 1204 includes thephysical structures in the building, such as walls, floors, ceilings,furniture, etc., all of which can absorb or give off heat. The buildingmass 1204 has a temperature T_(m) and a lumped mass thermal capacitanceC_(m). The resistance of the building mass 1204 to exchange heat withthe indoor air 201 (e.g., due to insulation, thickness/layers ofmaterials, etc.) may be characterized as mass thermal resistance R_(mi).

The outdoor air 1206 contributes outside air heat transfer {dot over(Q)}_(oa) to the data center 1200. The outdoor air 1206 is the airoutside of the building 10 with outdoor air temperature T_(oa). Theoutdoor air temperature T_(oa) fluctuates with the weather and climate.Barriers between the outdoor air 1206 and the indoor air 1201 (e.g.,walls, closed windows, insulation) create an outdoor-indoor thermalresistance R_(oi), to heat exchange between the outdoor air 1206 and theindoor air 1201.

The HVAC system 1299 also contributes heat to the data center 1200,denoted as {dot over (Q)}_(HVAC). The HVAC system 1299 includes HVACequipment 1210, controller 1212, an indoor air temperature sensor 1214and an outdoor air temperature sensor 1216, as well as a data equipmentheat predictor circuit 300. The HVAC equipment 1210 may include anysuitable equipment for controllably supplying heating and/or cooling tothe data center 1200. In general, HVAC equipment 1210 is controlled by acontroller 1212 to provide heating (e.g., positive value of {dot over(Q)}_(HVAC)) or cooling (e.g., a negative value of ({dot over(Q)}_(HVAC)) to the data center 1200.

The indoor air temperature sensor 1214 is located in the data center1200, measures the indoor air temperature T_(ia), and provides themeasurement of T_(ia) to the controller 1212. The outdoor airtemperature sensor 1216 is located outside of the building, measures theoutdoor air temperature T_(oa), and provides the measurement of T_(oa)to the controller 1212.

The data equipment heat predictor circuit 300 is configured to quantify,model, and/or predict the heat transfer {dot over (Q)}_(DCE) generatedby the data center equipment 1202. In various embodiments, the dataequipment heat predictor 300 follows various approaches calculating heatgenerated by computing devices. For example, in some embodiments arelationship between the complexity of calculations executed on the datacenter equipment 1202 and the heat generated by the data centerequipment is determined. Heat generation can then be modelled andpredicted based on predicted and/or schedule upcoming calculationsassigned to the data center equipment 1202.

In some embodiments, the data equipment heat predictor circuit 300follows a mechanism of communication approach. In such embodiments, thedata equipment heat predictor circuit 300 stores a model of a physicallayout of the data equipment. The data equipment heat predictor circuit300 then predicts each data transmission (or a statistical approximationthereof) through the data equipment based on a understanding of thestarting points, ending points, and paths taken therebetween of varioustransmissions, calculations, messages, etc. The distance of eachtransmission may be correlated to an amount of heat generated based on arelationship between electrical resistance, current, and heat. The totalamount of heat generated may then be calculated based on an aggregationof the heat generated by each communication/data transmission within thedata equipment 1202. In some embodiments, the data equipment heatpredictor circuit 300 may be configured to categorize portions of theheat generated by task, function, operation, time of day, day of week,season, holiday/workday, etc. in order to facilitate discrete controldecisions to manage the heat generated by the data equipment 1202.

The controller 1212 receives the temperature measurements T_(oa) andT_(ia) and a data equipment predicted heat, generates a control signalfor the HVAC equipment 1210, and transmits the control signal to theHVAC equipment 1210. The operation of the controller 1212 is discussedin detail below. In general, the controller 1212 considers the effectsof the data center equipment 1202, building mass 1204, and outdoor air1206 on the indoor air 1201 in controlling the HVAC equipment 1210 toprovide a suitable level of {dot over (Q)}_(HVAC). A model of thissystem for use by the controller 1212 is described with reference toFIG. 3 .

In the embodiments described herein, the control signal provide to theHVAC equipment 1210 by the controller 1212 indicates a temperaturesetpoint T_(sp) for the data center 1200. To determine the temperaturesetpoint T_(sp), the controller 1212 assumes that the relationshipbetween the indoor air temperature T_(ia) and the temperature setpointT_(sp) follows a proportional-integral control law with saturation,represented as:

{dot over (Q)} _(HVAC,j) =K _(p,j)ε_(sp) +K _(l,j)∫₀^(t)ε_(sp)(s)ds  (Eq. A)

ε_(sp) =T _(sp,j) −T _(ia)  (Eq. B)

where j ∈ {clg, hlg} is the index that is used to denote either heatingor cooling mode. Different parameters K_(p,j) and K_(l,j) are needed forthe heating and cooling mode. Moreover, the heating and cooling load isconstrained to the following set: {dot over (Q)}_(HVAC,j) ∈ [0, {dotover (Q)}_(clg,max)] for cooling mode (j=clg) and {dot over(Q)}_(HVAC,j) ∈ [−{dot over (Q)}_(htg,max),0] for heating mode (j=htg).As discussed in detail below with reference to FIG. 4 , the controller212 uses this model in generating a control signal for the HVACequipment 210.

Referring now to FIG. 13 , a circuit-style diagram 1300 corresponding tothe data center 1200 and the various heat transfers {dot over (Q)} ofFIG. 12 is shown, according to an exemplary embodiment. In general, thediagram 1300 models the data center 1200 as a control-oriented thermalmass system having two thermal resistance terms and two thermalcapacitance terms. This model can be characterized by the followingsystem of linear differential equations, described with reference toFIG. 13 below:

$\begin{matrix}{{C_{ia}{\overset{˙}{T}}_{ia}} = {{\frac{1}{R_{mi}}\left( {T_{m} - T_{ia}} \right)} + {\frac{1}{R_{oi}}\left( {T_{oa} - T_{ia}} \right)} - {\overset{˙}{Q}}_{HVAC} + {\overset{˙}{Q}}_{DCE}}} & \left( {{Eq}.C} \right)\end{matrix}$ $\begin{matrix}{{C_{m}{\overset{˙}{T}}_{m}} = {\frac{1}{R_{mi}}\left( {T_{ia} - T_{m}} \right)}} & \left( {{Eq}.D} \right)\end{matrix}$

where the first line (Eq. C) focuses on the indoor air temperatureT_(ia), and each term in Eq. C corresponds to a branch of diagram 1300as explained below:

Indoor air node 1302 corresponds to the indoor air temperature T_(ia).From indoor air node 1302, the model branches in several directions,including down to a ground 304 via a capacitor 1306 with a capacitanceC_(ia). The capacitor 1306 models the ability of the indoor air toabsorb or release heat and is associated with the rate of change of theindoor heat transfer {dot over (T)}_(ia). Accordingly, the capacitor1306 enters Eq. C on the left side of the equation as C_(ia)T_(ia).

From indoor air node 1302, the diagram 1300 also branches left tobuilding mass node 1310, which corresponds to the thermal masstemperature T_(m). A resistor 1312 with mass thermal resistance R_(mi)separates the indoor air node 1302 and the building mass node 1310,modeling the heat transfer {dot over (Q)}_(m) from the building mass1204 to the indoor air 1201 as

${\frac{1}{R_{mi}}\left( {T_{m} - T_{ia}} \right)}.$

This term is included on the right side of Eq. C above as contributingto the rate of change of the indoor air temperature {dot over (T)}_(ia).

The diagram 1300 also branches up from indoor air node 1302 to outdoorair node 1314. A resistor 1316 with outdoor-indoor thermal resistanceR_(oi) separates the indoor air node 302 and the outdoor air node 1314,modeling the flow heat from the outdoor air 1206 to the indoor air 1201as

$\frac{1}{R_{oi}}{\left( {T_{oa} - T_{ia}} \right).}$

This term is also included on the right side of Eq. C above ascontributing to the rate of change of the indoor air temperature {dotover (T)}_(ia).

Also from indoor air node 1302, the diagram 1300 branches right to two{dot over (Q)} sources, namely {dot over (Q)}HVAC and {dot over(Q)}_(DCE). As mentioned above, {dot over (Q)}_(DCE) corresponds to heatgenerated by the data center equipment 1202 that may have a significantimpact the rate of change of the indoor air temperature T_(ia). {dotover (Q)}_(DCE) is typically not measured by the HVAC system 1299, butmay be predicted and/or otherwise quantified/approximated by the dataequipment heat predictor. In some embodiments, the controller 1212 maybe configured to alter operation of the data center equipment 1202 tocontrol the amount of heat generated by the data center equipment 1202.{dot over (Q)}_(HVAC) is generated and controlled by the HVAC system 100to manage the indoor air temperature T_(ia). Accordingly, {dot over(Q)}_(HVAC) and {dot over (Q)}_(DCE) are included on the right side ofEq. C above. In some embodiments, an additional heat disturbance term{dot over (Q)}_(OTHER) is also included to model other heat disturbances(e.g., due to occupancy, solar irradiance, etc.).

The second nonlinear differential equation (Eq. D) above focuses on therate of change {dot over (T)}_(m) in the building mass temperature T.The capacity of the building mass to receive or give off heat ismodelled by capacitor 1318. Capacitor 1318 has lumped mass thermalcapacitance C_(m) and is positioned between a ground 1304 and thebuilding mass node 1310 and regulates the rate of change in the buildingmass temperature T_(m). Accordingly, the capacitance C_(m) is includedon left side of Eq. D. Also branching from the building mass node 1310is resistor 1312 leading to indoor air node 1302. As mentioned above,this branch accounts for heat transfer {dot over (Q)}m between thebuilding mass 1204 and the indoor air 1201. Accordingly, the term1/R_(mi)(T_(ia)−T_(m)) is included on the right side of Eq. D.

The model represented by diagram 1300 is used by the controller 1212 ingenerating a control signal for the HVAC equipment 1210. Moreparticularly, the controller 1212 uses a state-space representation ofthe model shown in diagram 1300. The state-space representation used bythe controller 1212 can be derived by incorporating Eq. A and B with Eq.C and D, and writing the resulting system of equations as a linearsystem of differential equations to get:

$\begin{matrix}{\begin{bmatrix}{\overset{˙}{T}}_{ia} \\{\overset{˙}{T}}_{m} \\\overset{.}{I}\end{bmatrix} = {\begin{bmatrix}{\frac{1}{C_{ia}}\left( {K_{p,j} - \frac{1}{R_{mi}} - \frac{1}{R_{oi}}} \right)} & \frac{1}{C_{ia}R_{mi}} & \frac{K_{I,j}}{C_{ia}} \\\frac{1}{C_{ia}R_{mi}} & {- \frac{1}{C_{ia}R_{mi}}} & 0 \\{- 1} & 0 & 0\end{bmatrix}{{{\begin{bmatrix}T_{ia} \\T_{m} \\I\end{bmatrix} + {\begin{bmatrix}{- \frac{K_{p,j}}{C_{ia}}} & \frac{1}{C_{ia}R_{oi}} \\0 & 0 \\1 & 0\end{bmatrix}\begin{bmatrix}T_{spj} \\T_{oa}\end{bmatrix}} + {\begin{bmatrix}0 \\\frac{1}{C_{ia}} \\0\end{bmatrix}{\overset{˙}{Q}}_{DCE}}};}}}} & \left( {{Eq}.E} \right)\end{matrix}$ $\begin{matrix}{{\begin{bmatrix}T_{ia} \\{\overset{˙}{Q}}_{{HVAC},j}\end{bmatrix} = {{\begin{bmatrix}1 & 0 & 0 \\{- K_{p,j}} & 0 & K_{I,j}\end{bmatrix}\begin{bmatrix}T_{ia} \\T_{m} \\I\end{bmatrix}} + {\begin{bmatrix}0 & 0 \\K_{p,j} & 0\end{bmatrix}\begin{bmatrix}T_{{sp},j} \\T_{oa}\end{bmatrix}}}};} & \left( {{Eq}.F} \right)\end{matrix}$

where I represents the integral term ∫₀ ^(t)ε_(sp)(s) ds from Eq. A. Theresulting linear system has three states (T_(ia), T_(m), I), two inputs(T_(sp, j), T_(oa)), two outputs (T_(ia), {dot over (Q)}_(HVAC)), andone disturbance {dot over (Q)}_(DCE).

In some cases, {dot over (Q)}_(DCE) is not measured or controlled andthe controller 212 models the disturbance {dot over (Q)}_(DCE) using aninput disturbance model that adds a forth state d to the state spacerepresentation. In a more compact form, this linear system ofdifferential equations can be written as:

$\begin{matrix}{{{\overset{˙}{x}(t)} = {{{A_{c}(\theta)}{x(t)}} + {{B_{c}(\theta)}{u(t)}}}};} & \left( {{Eq}.G} \right)\end{matrix}$ $\begin{matrix}{{{y(t)} = {{{C_{c}(\theta)}{x(t)}} + {{D_{c}(\theta)}{u(t)}}}};} & \left( {{Eq}.H} \right)\end{matrix}$ where ${{A_{c}(\theta)} = \begin{bmatrix}{- \left( {\theta_{1} + \theta_{2} + {\theta_{3}\theta_{4}}} \right)} & \theta_{2} & {\theta_{3}\theta_{4}\theta_{5}} \\\theta_{6} & {- \theta_{6}} & 0 \\{- 1} & 0 & 0\end{bmatrix}},{{B_{c}(\theta)} = \begin{bmatrix}{\theta_{3}\theta_{4}} & \theta_{1} \\0 & 0 \\1 & 0\end{bmatrix}},$ ${{C_{c}(\theta)} = \begin{bmatrix}1 & 0 & 0 \\{- \theta_{4}} & 0 & {\theta_{5}\theta_{4}}\end{bmatrix}},{{{D_{c}(\theta)} = \begin{bmatrix}0 & 0 \\\theta_{4} & 0\end{bmatrix}};}$${\theta_{1} = \frac{1}{C_{ia}R_{oi}}};{\theta_{2} = \frac{1}{C_{ia}R_{mi}}};{\theta_{3} = \frac{1}{C_{ia}}};{\theta_{4} = K_{p}};{\theta_{5} = \frac{1}{\tau}};{\theta_{6} = \frac{1}{C_{m}R_{mi}}};{and}$${{\overset{˙}{x}(t)} = \begin{bmatrix}{\overset{˙}{T}}_{ia} \\{\overset{˙}{T}}_{m} \\\overset{.}{I}\end{bmatrix}};{{x(t)} = \begin{bmatrix}T_{ia} \\T_{m} \\I\end{bmatrix}};{{u(t)} = {\begin{bmatrix}T_{spj} \\T_{oa}\end{bmatrix}.}}$

In some embodiments, the controller 212 uses a two-step process toparameterize the system. In the first step, the controller 212identifies the system parameters θ={θ1, θ2, θ3, θ4, θ5, θ6} (i.e., thevalues of C_(ia), C_(m), R_(mi), R_(oi), K_(p,j), K_(i,j)). Thedisturbance state d is then introduced into the model and an Kalmanestimator gain is added, such that in the second step the controller 212identifies the Kalman gain parameters K.

In some embodiments, the disturbance state d may be modelled based onpredictions generated by the data equipment heat predictor circuit 300to facilitate system identification. In other embodiments, the dataequipment heat predictor circuit 300 may provide predictions/model of{dot over (Q)}_(DCE) that allows the system to be model without the needfor the disturbance state approach.

As used herein, the term ‘variable’ refers to an item/quantity capableof varying in value over time or with respect to change in some othervariable. A “value” as used herein is an instance of that variable at aparticular time. A value may be measured or predicted. For example, thetemperature setpoint T_(sp) is a variable that changes over time, whileT_(sp)(3) is a value that denotes the setpoint at time step 3 (e.g., 68degrees Fahrenheit). The term “predicted value” as used herein describesa quantity for a particular time step that may vary as a function of oneor more parameters. In various embodiments, time steps have variousintervals therebetween (e.g., one minute, fifteen minutes, one hour,etc.).

Controller for HVAC Equipment with System Identification

Referring now to FIG. 14 , a detailed diagram of the controller 1212 isshown, according to an exemplary embodiment. The controller 1212includes a processing circuit 1400 and a communication interface 1402.The communication interface 1402 is structured to facilitate theexchange of communications (e.g., data, control signals) between theprocessing circuit 1400 and other components of HVAC system 100. Asshown in FIG. 14 , the communication interface 1402 facilitatescommunication between the processing circuit 1400 and the outdoor airtemperature sensor 1216 and the indoor air temperature sensor 1214 toall temperature measurements T_(oa) and T_(ia) to be received by theprocessing circuit 1400. The communication interface 1402 alsofacilitates communication between the processing circuit 1400 and theHVAC equipment 1210 that allows a control signal (indicated astemperature setpoint T_(sp)) to be transmitted from the processingcircuit 1400 to the HVAC equipment 1210.

The processing circuit 1400 is structured to carry out the functions ofthe controller described herein. The processing circuit 1400 includes aprocessor 1404 and a memory 1406. The processor 1404 may be implementedas a general-purpose processor, an application-specific integratedcircuit, one or more field programmable gate arrays, a digital signalprocessor, a group of processing components, or other suitableelectronic processing components. The memory 1406, described in detailbelow, includes one or more memory devices (e.g., RAM, ROM, NVRAM, FlashMemory, hard disk storage) that store data and/or computer code forfacilitating at least some of the processes described herein. Forexample, the memory 1406 stores programming logic that, when executed bythe processor 1404, controls the operation of the controller 212. Moreparticularly, the memory 1406 includes a training data generator 1408, atraining data database 1410, a model identifier 1412, a model predictivecontroller 1414, and an equipment controller 1416. The variousgenerators, databases, identifiers, controllers, etc. of memory 1406 maybe implemented as any combination of hardware components andmachine-readable media included with memory 1406.

The equipment controller 1416 is configured to generate a temperaturesetpoint T_(sp) that serves as a control signal for the HVAC equipment1210. The equipment controller receives inputs of the indoor airtemperature T_(ia) from the indoor air temperature sensor 1214 via thecommunication interface 1402 and {dot over (Q)}_(HVAC) from the modelpredictive controller 1414 (during normal operation) and the trainingdata generator 1408 (during a training data generation phase describedin detail below). The equipment controller uses T_(ia) and {dot over(Q)}_(HVAC) to generate T_(sp) by solving Eq. A and Eq. B above forT_(sp). The equipment controller 1416 then provides the control signalT_(sp) to the HVAC equipment 1210 via the communication interface 1402.

The model predictive controller 1414 determines {dot over (Q)}_(HVAC)based on an identified model and the temperature measurements T_(ia),T_(oa), and provides {dot over (Q)}_(HVAC) to the equipment controller1416. The model predictive controller 1414 follows a model predictivecontrol (MPC) approach. The MPC approach involves predicting futuresystem states based on a model of the system, and using thosepredictions to determine the controllable input to the system (here,{dot over (Q)}_(HVAC)) that bests achieves a control goal (e.g., tomaintain the indoor air temperature near a desired temperature). A moreaccurate model allows the MPC to provide better control based on moreaccurate predictions. Because the physical phenomena that define thebehavior of the system (i.e., of the indoor air 201 in the building 10)are complex, nonlinear, and/or poorly understood, a perfect modelderived from first-principles is generally difficult. Thus, the modelpredictive controller 1414 uses a model identified through a systemidentification process facilitated by the training data generator 1408,the training data database 1410, the model identifier 1412, and the dataequipment heat predictor circuit 300 as described in detail below.

System identification, as facilitated by the training data generator1408, the training data database 1410, data equipment heat predictorcircuit 300, and the model identifier 1412, is a process of constructingmathematical models of dynamic systems. System identification provides asuitable alternative to first-principles-derived model when firstprinciples models are unavailable or too complex for on-line MPCcomputations. System identification captures the important and relevantsystem dynamics based on actual input/output data (training data) of thesystem, in particular by determining model parameters particular to abuilding or zone to tune the model to the behavior of the building/zone.As described in detail below, the training data generator 1408, thetraining data database 1410, the data equipment heat predictor circuit300 and the model identifier 1412 each contribute to systemidentification by the controller 1212.

The training data generator 1408 is configured to generate training databy providing an excitation signal to the system. That is, the trainingdata generator provides various {dot over (Q)}_(HVAC) values to theequipment controller 1416 for a number N of time steps k, and receivesthe measured output response of the indoor air temperature T_(ia) ateach time step k from the air temperature sensor 1214. The various {dotover (Q)}_(HVAC) values may be chosen by the training data generator1408 to explore the system dynamics as much as possible (e.g., across afull range of possible {dot over (Q)}_(HVAC) values, different patternsof {dot over (Q)}_(HVAC) values, etc.).

The equipment controller 1416 receives the various {dot over (Q)}_(HVAC)values and generates various control inputs T_(sp) in response. Thetemperature setpoint T_(sp) for each time step k is provided to the HVACequipment 1210, which operates accordingly to heat or cool the zone 200(i.e., to influence T_(ia)). The temperature setpoints T_(sp) may alsobe provided to the training data generator 1408 to be included in thetraining data. The training data generator receives an updatedmeasurement of the indoor air temperature T_(ia) for each time step kand may also receive the outdoor air temperature T_(oa) for each timestep k. The training data generator 1408 thereby causes the states,inputs, and outputs of the system to vary across the time steps k andgenerates data corresponding to the inputs and outputs.

The inputs and outputs generated by the training data generator 1408 areprovided to the training data database 1410. More particularly, in thenomenclature of the model of Eq. E and Eq. F above, the training datagenerator 1408 provides inputs T_(sp) and T_(oa) and outputs {dot over(Q)}_(HVAC) and T_(ia) for each time step k to the training datadatabase 1410.

The training data database 1410 stores the inputs and outputs for eachtime step k provided by the training data generator 1408. Each input andoutput is tagged with a time step identifier, so that data for the sametime step can be associated together. The training data database 1410thereby collects and stores input and output data for each time step k,k=0, N, or, more specifically, T_(sp)(k), T_(oa)(k), T_(ia)(k), and {dotover (Q)}_(HVAC)(k), for k, k=0, . . . , N. This data is groupedtogether in the training data database 410 in a set of training dataZ^(N). In the notation of Eq. G and Eq. H, Z^(N)=[y(1), u(1), y(2),u(2), . . . , y(N), u(N)].

In some embodiments, the training data is refined using a saturationdetection and removal process. System and methods for saturationdetection and removal suitable for use to refine the training data Z^(N)are described in U.S. patent application Ser. No. 15/900,459, filed Feb.20, 2018, incorporated by reference herein in its entirety. For example,as described in detail therein, the training data may be filtered bydetermining whether the operating capacity is in a non-transient regionfor a threshold amount of a time period upon determining that an errorfor the building zone exists for the time period, and in response to adetermination that the operating capacity is in the non-transient regionfor at least the threshold amount of the time period, indicating thetime period as a saturation period. Data from the saturation period canthen be removed from the training data.

The model identifier 1412 accesses the training data database 410 toretrieve the training data Z^(N) and uses the training data Z^(N) toidentify a model of the system. The model identifier 1412 includes asystem parameter identifier 1418 and a gain parameter identifier 1420.The system parameter identifier 1418 carries out a first step of systemidentification, namely identifying the model parameters, while the gainparameter identifier 1420 carries out a second step, namely determininga Kalman gain estimator. This two-step process is described in detail inU.S. patent application Ser. No. 15/953,324, filed Apr. 13, 2018,incorporated by reference herein in its entirety.

The model parameters and the Kalman gain estimator are included in anidentified model of the system, and that model is provided to the modelpredictive controller 1414. The model predictive controller can thusfacilitate the control of the HVAC equipment 1210 as described above.

In some embodiments, the data equipment heat predictor circuit 300provides predictions, models, and/or other data relating to the heatgenerated by the data center equipment 1202 to the model identifier1412. The model identifier 1412 may use such data in identifying modelparameters and/or determining a Kalman gain estimator. For example, themodel identifier 1412 may use data from the data equipment heatpredictor circuit 300 to identify trends in the training data Z^(N)attributable to operation of the data center equipment 1202. The systemmodel identified by the model identifier 1412 may then be tailored toinclude an understanding of the influence of the heat Q_(DCE) providedby the data center equipment 1202. For example, a model may be generatedthat uses predictions and/or other data from the data equipment heatpredictor circuit 300 as an input to the system model.

In some cases, the model predictive controller 1414 receives a dataequipment predicted heat, a time series of data equipment predictedheats, etc. from the data equipment heat predictor circuit 300 and usesthose predictions in optimizing control of the HVAC equipment 1210. Forexample, the model predictive controller 1414 may reduce the temperaturesetpoint in advance of a period of predicted high data equipmentactivity to pre-cool the data center 1200, either by increasing {dotover (Q)}_(HVAC) (when the T_(oa) is greater than T_(ia)) or reducing{dot over (Q)}_(HVAC) (when the T_(oa) is lower than T_(ia)). This mayreduce an overall cost of operating the HVAC equipment 1210 and minimizea demand charge over an optimization period.

Integrated Controller

Referring now to FIG. 15 , a block diagram of an integrated controller1500 is shown, according to an exemplary embodiment. The integratedcontroller 1500 is configured to coordinate operation of the data centerequipment 1202 and the HVAC equipment 1210 to achieve desiredenvironmental condition(s) for the data center 1200 at a minimize cost,for example to maintain the data center equipment 1202 within a desiredtemperature range while minimize utility consumption.

The data center equipment controller 1502 may be configured to makevarious control decisions regarding the operation of the data centerequipment 1202. For example, the data center equipment controller 1502may be configured to make scheduling decisions (i.e., to determine whento execute particular operations) and resource allocation decisions(i.e., to choose which device(s) of the data center equipment 1202 willexecute particular operations). As further examples, the data centerequipment controller 1502 may determine how many virtual machines tospool up, decide how much memory to allocate (e.g., allocating more RAMmemory to a processing intensive task to help reduce processing and harddrive heat), how much CPU utilization to utilize, data rates, line load,an amount of parallelization, etc. Accordingly, the data centerequipment controller 1502 is configured to make various controldecisions that affect the amount of heat generated by the data centerequipment 1202 and the schedule of the heat generate by the data centerequipment 1202.

As described above, the HVAC model predictive controller 1414 isconfigured to predict future system states, loads, and rates andgenerate controls for the HVAC equipment 1210 to optimize a costfunction associated with operating the HVAC equipment 1210. Accordingly,the HVAC model predictive controller 1414 is configured to control theamount of heat provided/removed by the HVAC equipment 1210 and theschedule of the heat provided/removed by the HVAC equipment 1210.

The coordinating agents 1504 are configured to facilitate coordinationbetween the data center equipment controller 1502 and the HVAC modelpredictive controller 1414. The coordinating agents 1504 may be based onone or more of a variety of artificial intelligence approaches, and mayinclude a provisioning agent, scheduling agent, thermal load agent, MPCagent, HVAC optimization agent, etc. The coordinating agents 1504 mayfacilitate the HVAC model predictive controller 1414 in controlling theHVAC equipment 1210 to anticipate various increases or decreases in heatgenerated by the data center equipment 1202. The coordinating agents1504 may also facilitate the data center equipment controller 1502 inprovisioning, scheduling, assigning, managing, etc. the data centerequipment 1202 and operations executed thereon to minimize an overallcost of operating the HVAC equipment.

For example, the coordinating agents 1504 may direct the data centerequipment controller 1502 to reduce the heat generated by the datacenter 1202 when the cost of operating the HVAC equipment 1210 is high(e.g., when a utility rate is high, when an outside air temperature ishigh), and to increase the operation of the data center 1202 when thecost of operating the HVAC equipment 1210 is low. As another example,the coordinating agents 1504 may control the data center equipment 1202to generate extra heat (e.g., beyond what may be necessary for ordinaryoperation) to reducing a heating load on the HVAC equipment 1210. Asanother example, the coordinating agents 1504 may monitor health, risk,bugs, bad reads, errors that result in recalculations or re-reads,corrupted sectors, fragmentations of hard drives, HVAC equipment faults,etc. and coordinate control of the data center equipment 1202 and HVACequipment 1210 to adjust for such events. The coordinating agents 1504may be configured to make many such coordination decisions.

Feedforward Control of Building Equipment Based on Per Server HeatContribution

Referring now to FIG. 16 , a block diagram of a system 1600 configuredto facilitate feedforward control of building equipment 1602 based onper server heat contribution in a data center is shown, according to anexemplary embodiment. As shown in FIG. 16 , the system 1600 includesmultiple servers, shown as server A 1604, server B 1606, through serverZ 1608. Any number of servers may be included in various embodiments.

Server A 1604 is shown to include a server heat contribution agent 1610configured to determine a heat contribution of server A. That is, theserver heat contribution agent 1610 of server A 1604 is configured todetermine the heat (e.g., total thermal energy) generated by server A1604 during operation of the server A 1604. The heat generated by theserver A 1604 varies based on the utilization of the server A 1604(e.g., calculations performed, data transmitted, programs executed,etc.). The server heat contribution agent 1610 for server A may use anartificial intelligence or rules-based algorithm to determine a heatoutput of server A 1604. In some embodiments, the server heatcontribution agent 1610 determines the heat output of server A 1604 inreal-time (i.e., for a current/ongoing time period). In otherembodiments, the server heat contribution agent 1610 predicts the heatoutput of server A 1604 over a future time period, for example based onhistorical data and/or queued tasks awaiting execution by the server A1604. Various techniques can be used to determine heat output including,but not limited to, assessing processor utilization, clock speeds,memory usage, operations per unit time, power supply output, etc.

The various other servers (e.g., server B 1606 and server Z 1608) alsoinclude heat contribution agents (shown in FIG. 16 as server heatcontribution agent 1612 and server heat contribution agent 1614). Theheat generated by each of the servers 1604-1608 is thereby ascertainedon a per server basis (i.e., independently for each server 1604-1608).In other embodiments, heat contribution agents are associated with eachCPU, each blade of a data server, or some other sub-unit of a datacenter, and are configured to ascertain the heat generated by thecorresponding sub-unit.

The server heat contribution agents 1610-1614 provide the calculatedvalues of heat contribution for the corresponding servers to a datacenter heat aggregation circuit 1614. The data center heat aggregationcircuit 1614 is configured to receive the values of the heatcontribution of the servers 1604-1608 and aggregate (e.g., sum) thevalues to determine a total amount of heat generated by the servers1604-1608. The data center heat aggregation circuit 1614 provides anindication of the total amount of heat generated by the servers1604-1608 to an equipment controller 1616.

The equipment controller 1616 is configured to receive the indication ofthe total amount of heat generated by the servers 1604-1608 and performa feedforward control process based on the indication from the datacenter aggregation circuit 1614. The equipment controller 1616 isconfigured to determine one or more setpoints for the building equipment1602 that cause the building equipment 1602 to operate to offset theamount of heat generated by the servers 1604-1608, thereby maintainingthe data center at a desired indoor air temperature (e.g., an optimaltemperature for operation of the servers 1604-1608). For example, theequipment controller 1616 may control the building equipment 1602 toremove an amount of heat from the data center approximately equal to anamount of heat generated by the servers 1604-1608. In some embodiments,the equipment controller 1616 also accounts for the effects of outdoorair temperature, solar irradiance, etc. on the indoor air temperature,for example increasing the amount of heat removed from the data centerby the building equipment 1602 in a scenario where the outdoor airtemperature is higher than the desired temperature of the data center.

The amount of heat generated by the servers 1604-1608 (or othersub-units of data center in various embodiments) is thereby ascertainedon a per-server (or per-sub-unit) basis and fed forward into a controlalgorithm for building equipment 1602. The building equipment 1602 canthus be controlled to compensate for server heat generation, withoutwaiting for deviations in measured air temperature from a setpoint andthen correcting for such deviations as in other temperature controlapproaches. A more consistent air temperature at the data center maythus be achieved.

In some embodiments, the heat contribution agent predicts the heatcontribution for each respective server 1604, 1606, and 12608 based uponone or more of time of day, time of year, operation schedules,maintenance schedules, historical data, da of the week, etc. Thepredicted heat load can be provided for future time intervals asappropriate. Pre-cooling, feedforward control, and/or optimization canbe performed using the predicted heat load on an individual serverbasis, in some embodiments.

Predictive Control for Data Center

Referring now to FIG. 17 , a flowchart of a process 1700 for modelpredictive control for a data center is shown, according to an exemplaryembodiment. The process 1700 can be executed using the system 1299, forexample. In various embodiments, the process 1700 is used for control ofHVAC systems, variable refrigerant flow (VRF) systems, in-floor coolingsystems, rooftop units, data center rack cooling equipment, etc.

At step 1702, a model is provided that predicts thermal behavior of thedata center. The model uses an amount of thermal energy generated by thedata center equipment as an input. In some embodiments, the model usesdifferent amounts of thermal energy generated by different portions ofthe data center equipment as multiple inputs. In some embodiments, themodel is a grey-box model that can be identified using historical data,for example as described above with reference to FIGS. 12-14 . In otherembodiments, the model is implemented as a neural network or otherlearning-based or artificial intelligence algorithm. Parameters of theneural network may be obtained using machine learning, for examplesupervised or unsupervised learning. In some embodiments, a grey-boxmodel as described above is used to generate additional data tofacilitate training of a neural network, for example as described inU.S. patent application Ser. No. 16/413,946, filed May 16, 2019, theentire disclosure of which is incorporated by reference herein.

At step 1704, values of the amount of thermal energy generated by thedata center are predicted. For example, in some embodiments arelationship between the complexity of calculations executed on the datacenter equipment 1202 and the heat generated by the data centerequipment is determined. Heat generation can then be modelled andpredicted based on predicted and/or scheduled upcoming calculationsassigned to the data center equipment 1202.

As another example, in some embodiments, a model of a physical layout ofthe data equipment is used and each data transmission (or a statisticalapproximation thereof) through the data equipment is analyzed based on aunderstanding of the starting points, ending points, and paths takentherebetween of various transmissions, calculations, messages, etc. Thedistance of each transmission may be correlated to an amount of heatgenerated based on a relationship between electrical resistance,current, and heat. The total amount of heat generated may then becalculated based on an aggregation of the heat generated by eachcommunication/data transmission within the data equipment.

As another example, repeating patterns of the heat generated by the dataequipment may be ascertained, for example based on time-of-day,day-of-week, holiday/workday, etc. In some embodiments, amachine-learning algorithm is trained on historical data to predict theamount of heat that will be generated by data center equipment based oninputs such as day of the week, time of day, season, weather, economicconditions, special events, or other domain-specific factors (i.e.,relating to the field of use of the data center). Predictions of theamount of thermal energy generated by the data equipment may therebygenerated at step 1704.

At step 1706, setpoints for the building equipment are determined byperforming an optimization process using the predicted values and themodel. In some embodiments, the optimization process includesdetermining setpoints for the building equipment that are predicted tocause a temperature of the data center to be maintained within anacceptable range while minimizing an overall amount of energy usage, atotal operational cost, or some other metric. Some such optimizationapproaches are described above with reference to FIGS. 1-14 . In otherembodiments, an artificial intelligence agent is configured toautomatically select the setpoints. Such an artificial intelligenceagent can be trained on historical data to find setpoints that allow forminimization of overall amount of energy usage, a total operationalcost, or some other metric while conforming to temperature constraintsof the data center.

At step 1708, the building equipment is controlled in accordance withthe setpoints. For example, the setpoints may be amounts of thermalenergy to be added or removed from the data center by the buildingequipment. In such embodiments, the building equipment is controlled toadd or remove approximately the amounts of thermal energy defined by thesetpoints from the data center. As another example, the setpoints may beindoor air temperature setpoints for the data center, and the buildingequipment may be controlled using feedback control to drive the indoorair temperature towards the setpoints.

At step 1710, control of the data center equipment is coordinated withcontrol of the building equipment. Such coordination may be executed byartificial intelligence agents, for example coordinating agents 1504 ofFIG. 15 . For example, in some embodiments coordinating control of thedata center equipment with control of the building equipment may resultin shifting time-insensitive calculations to times when weather or otherfactors place a lower load on the building equipment, for example earlymorning when outdoor air temperatures are lowest. As another example,process 1700 may result in pre-cooling of a data center in advance oftimes of predicted or schedule high activity for the data center. Atstep 1710, operations of the data center equipment may be scheduledbased on a time-variant cost of operating the building equipment and/orthe data center equipment. For example, time-varying utility rates maybe accounted for in scheduling operation of the data center equipmentand the building equipment to minimize a total utility cost of operatingboth the data center equipment and the building equipment.

In some embodiments, the process 1700 is performed on a per-serverbasis, where the both the setpoints of the building equipment andoperating parameters of individual units of data center equipment arecontrolled using the model. For example, the model can control whichservers operate at which rates or execute which tasks to control heatgeneration in the data room, i.e., by the various servers. In someembodiments, more heat efficient servers or servers in cooler portionsof the data room are selected for more operations depending uponpredicted heat load. In another example, current cooling capacity on aper-server basis is determined and the load on the servers is adjustedto match the heat contribution to the local cooling capacity for eachserver. Advantageously, such a model may take advantage of data roomswhere certain areas, walls, sub-rooms, racks, etc. have higher coolingcapacity than other areas in the data center to optimize heatcontribution and control across all areas of the room. In someembodiments, the local cooling capacity is weather dependent and themodel accounts for weather with respect to local cooling capacity (e.g.,colder weather may result in greater cooling capacity at the externalwalls and hotter weather may result in less cooling capacity alongexternal walls). FIG. 23 provides an illustration relating to suchimplementations.

Coordinated Control of Building Equipment and Data Center Equipment

Referring now to FIGS. 18-19 , examples processes 1800 and 1900 ofcoordinated control of building equipment and data center equipment areshown. The processes 1800 and 1900 may occur as part of the process 1700of FIG. 17 , as a result of operation of various systems and methodsdescribed herein, or may be executed independently.

FIG. 18 shows a flowchart of a process 1800 for coordinated control ofbuilding equipment and data center equipment. At 1802, an upcominghigh-activity period for the data center is determined. In someembodiments, the high-activity period may be pre-scheduled, for examplewith a large task or large number of tasks planed for a particularupcoming time period. In other embodiments, the high-activity period ispredicted based on historical data, time of day, etc. For example, aspike in activity of the data center may occur every day at a similartime, for example at a beginning of business hours (e.g., for businessservices servers), in the evening (e.g., for a data center that handlesentertainment services such as video streaming), at an initial releaseof content to a large audience, etc.

At step 1804, the data center is pre-cooled in advance of the upcominghigh-activity period. For example, a temperature setpoint for the datacenter may be lowered to a bottom of an optimal temperature range forthe data center equipment or below such a temperature range. Buildingequipment (including, in some cases, specialty data center coolingdevices) may be controlled to drive the temperature of the data centerto the lowered setpoint, thereby reducing the temperature in the datacenter before the high-activity period.

When the high-activity period occurs, the data center equipment willgenerate heat. Because the data center was pre-cooled, the temperatureof the data center can be allowed to drift upwards during thehigh-activity period. The building equipment need not offset theentirety of the heat generation of the data center equipment during thehigh-activity period. The peak load on the building equipment maythereby be reduced. Furthermore, process 1800 may result in a separationof peak operation of the data center equipment from peak operation ofthe building equipment which can reduce peak demand on energy resourcesfor the overall facility.

FIG. 19 illustrates a process 1900 for coordinated control of buildingequipment and data center equipment. At step 1902, low-cost periodsand/or high-costs periods for operation of building equipment areidentified. In some embodiments, low-cost and high-cost periods aredifferentiated based on time varying utility rates, such that thelow-cost periods correspond to relatively cheap utility rates while thehigh-cost periods correspond to relative expensive utility rates. Insome embodiments, the low-cost and high-cost periods are defined basedon an incentive-based response program of a utility company, for examplea demand response program or a peak contribution program. In otherembodiments, the low-cost and high-cost periods are determined based onthe relative efficiency (e.g., coefficient of performance) of operatingthe building equipment at different times. For example, coolingequipment may operate more efficiently at lower outdoor airtemperatures, and in some cases outdoor air may be used directly to coolthe data center. Accordingly, the low-cost periods may be periods of lowoutdoor air temperature, where the high-cost periods are periods of highoutdoor air temperature.

At step 1904, time-insensitive data center tasks scheduled forhigh-costs periods are identified. For example, a list of all tasksscheduled to be executed by the data center during high-costs periodsmay be determined. Those tasks may then be classified based on the typeof task, due date for execution, criticality, priority, etc. Based onsuch criteria, those tasks which are time-insensitive (i.e., which donot need to be executed at a particular time or during the high-costperiod) are identified.

At step 1906, data center operations are rescheduled to shift the tasksidentified at step 1904 to low-cost periods. That is, step 1906 includesactively managing the data center schedule and operation to cause tasksto be executed during low-cost periods where possible. As a result, heatgeneration associated with such tasks can be compensated for by thebuilding equipment during low-cost periods rather than during high-costperiods, thereby reducing the overall cost of operating the buildingequipment and the data center facility as a whole.

Total Resource Consumption Optimization with Server Efficiency Model

Referring now to FIG. 20 , a flowchart of a process 2000 for optimizingresource consumption of a data center is shown, according to anexemplary embodiment. As described in detail in the following, theprocess 2000 includes consideration of the resource consumption ofbuilding equipment, the resource consumption of the data centerequipment, and the efficiency of the data center equipment as a functionof temperature. By accounting for these characteristics, control of thebuilding equipment and the data center equipment can be optimized tominimize resource consumption while meeting performance expectations forthe data center equipment.

At step 2002, an objective function is defined that represents a totalresource consumption of a data center facility. The objective functionmay account for both resource consumption by building equipment andresource consumption by data center equipment. For example, in someembodiments, the objective function takes the form of: J(x, y, t)=J_(BE)(x, t)+J_(DCE)(y, t), where J_(BE)(x, t) describes a total resourceconsumption of the building equipment at time t for a control dispatchx, while J_(DCE)(y, t) describes a total resource consumption of thedata center equipment at time t for a control variable y. The controlvariable x may be defined variously depending on the building equipmentavailable in a given embodiment, and may include temperature setpoints,energy targets, operating setpoints, etc. in various embodiments. Thecontrol variable y for the data center may be defined variouslydepending on the controllable variables relating to operation of thedata center equipment. For example, in some embodiments, operation ofthe data center equipment can be characterized based on a number oftasks y performed by the data center equipment in a time step. Forexample, the resource consumption may be defined as J_(DCE)(y,t)=ρ(t)*y(t), where ρ(t) defines an amount of resources consumed pertask and y(t) is a number of tasks performed by the data centerequipment at time t.

At step 2004, a first constraint is defined based on a predictivethermal model of the data center. For example, the predictive thermalmodel may be defined as described above with reference to FIG. 13 andidentified based on training data as described with reference to FIG. 14. For example, as explained in further detail above, the thermal modelmay be defined as:

${C_{ia}{\overset{˙}{T}}_{ia}} = {{\frac{1}{R_{mi}}\left( {T_{m} - T_{ia}} \right)} + {\frac{1}{R_{oi}}\left( {T_{oa} - T_{ia}} \right)} - {\overset{˙}{Q}}_{HVAC} + {\overset{˙}{Q}}_{DCE}}$${C_{m}{\overset{˙}{T}}_{m}} = {\frac{1}{R_{mi}}{\left( {T_{ia} - T_{m}} \right).}}$

In this example, {dot over (Q)}_(HVAC) is a function of the controlvariable x and {dot over (Q)}_(DCE) is a function of the controlvariable y, thereby creating interactions between the objective functionand the thermal model. In other embodiments, other types of models, forexample a neural-network-based model may be used to predict thermalbehavior of the data center. In such an example, the neural network mayuse the variables x and y as inputs to the neural network.

At step 2006, a second constraint is defined based on atemperature-dependent efficiency of the data center equipment. Forexample, the value of the efficiency function ρ(t) may be defined as afunction of air temperature at the data center equipment, i.e., indoorair temperature T_(ia), for example ρ(t)=ρ(T_(ia)(t)). In such a case,the per-task resource consumption is modelled as being dependent on theair temperature at the data center equipment. As one possible example,the efficiency function ρ(T_(ia)(t)) may be defined as a Gaussian curvewith a maximum values at an optimal temperature for the data centerequipment and decreasing away from such optimal temperature. Theefficiency function ρ(t) may be dependent on various other variables invarious embodiments.

At step 2008, a third constraint is defined based on the tasks to beexecuted by the data center equipment. For example, due to expectationsof the performance of the data center equipment, the values of y(t) maybe required to exceed a minimum value. For example, the total number oftasks over a time period including multiple time steps may be requiredto exceed a minimum value, i.e., ∫_(t=0) ^(t=T)y(t)y_(min,total). Insuch a case, the third constraint allows a degree of freedom inallocation of the tasks across the time period (i.e., across t=0, . . .T).

In some embodiments, additional constraints are imposed. For example, amaximum number of tasks for any particular time may be imposed (e.g.,y(t)<y_(max)) based on limitations of the data center equipment.

At step 2010, the objective function is optimized subject to theconstraints to determine setpoints and/or schedules for the buildingequipment and/or the data center equipment. For example, an optimizationprocess may be performed to select values of the number of calculationsy to be performed by the data center equipment and values of thedispatch x (e.g., an energy target, a temperature setpoint, an operatingsetpoint) for the building equipment which, together, correspond to aminimum value of the resource consumption of the data center whilesatisfying the constraints. In some examples, the optimization processis optimized over a time horizon, for example, to select the minimumvalue of ∫_(t=0) ^(t=T) J(x, y, t), for example by optimally allocatingdata center computation tasks and building equipment operation acrossthe time steps t of period t=0, . . . , T. Resource consumption for theentire data center can thereby be optimized in an integrated process.

The notation and formulation of FIG. 20 relates to some embodiments ofthe process 2000. Other formulations and algorithms are also possible,for example based on an artificial intelligence, neural network, and/ormachine learning approach to optimizing overall resource usage of thedata center.

Asset Allocation System for Data Center

Referring now to FIG. 21 , a block diagram of a system 2100 for managingoperation of a data center is shown, according to an exemplaryembodiment. The system 2100 includes an asset allocation server 2102,central plant equipment 2104, airside system 2106, computer room airconditioners 2108, rack cooling units 2110, data center equipment 1202,and sensors 2112. The system 2100 also includes controllers 2114configured to control the central plant equipment 2104, airside system2106, computer room air conditioners 2108, and rack cooling units 2110.In other embodiments, other heating and/or cooling equipment is include(e.g., variable refrigerant flow systems, liquid cooling systems,rooftop units, etc.). The data center equipment 1202 is shown to includeserver agents 2116, server fans 2118, and sensors 2120.

The central plant equipment 2104, airside system 2106, computer room airconditioners 2108, rack cooling units 2110, and server fans 2118 allfunction to affect the temperature of the data center equipment. Thecentral plant equipment 2104 may include subplants described withreference to FIGS. 1-10 . The airside system 2106 includes HVAC airhandling units and/or other airside equipment that uses chilled water(or hot water) from the central plant equipment 2104 to heat or cool abuilding zone.

The computer room air conditioners 2108 may be specialized coolingequipment for a data room and can be rooftop units. The computer roomair conditioners 2108 may include local refrigeration equipment or mayuse chilled water from the central plant equipment 2104 (or otherresource such as a variable refrigerant flow outdoor unit). The computerroom air conditioners 2108 are positioned in the data center and removeheat from the data center. In some embodiments, the computer room airconditioners 2108 intake warm ambient air from the data center, cool theair using a chilled fluid (from the central plant equipment 2104 orchilled locally at the computer room air conditioner 2108), and providethe cooled air into sub-floor ducts. The cooled air moves through thesub-floor ducts, which vent out at locations aligned with server racksfor the data center equipment 1202. The computer room air conditioners2108 thereby operated to move cooled air directly to the data centerequipment 1202.

The rack cooling units 2110 are installed on the server racks (i.e.,mounted on a shared structure with the data center equipment 1202) andoperated to cool the data center equipment 1202. In some embodiments,the rack cooling units 2110 receive chilled fluid from the central plantequipment 2104 (or other resource such as a variable refrigerant flowoutdoor unit). In other embodiments, a refrigeration cycle is executedlocally at the rack cooling units 2110. The rack cooling units 2110operate locally at individual units of data center equipment 1202 andmay therefore be advantageous for targeting temperature changes atindividual units of data center equipment 1202

The server fans 2118 are located on the servers (data center equipment1202) and operate to blow air across the servers to cool the servers.The effectiveness of the fans 2118 is dependent on the temperature ofthe air taken in by the fans 2118 to be blown across the servers. Theserver fans 2118 may consume electricity. The server fans 2118 can becontrolled by the asset allocation server 2102 by communicating withserver agents 2116.

The asset allocation server 2102 is configured to allocate loads acrossthe central plant equipment 2104, airside system 2106, computer room airconditioners 2108, rack cooling units 2110, and server fans 2118, forexample to optimally manage the temperature of the data center equipment1202. In some embodiments, the asset allocation server 2102 isconfigured to coordinate control of or assist coordination of thecentral plant equipment 2104, airside system 2106, computer room airconditioners 2108, rack cooling units 2110, and server fans 2118. Theoverall thermal behavior of the system can be optimized in an integratedprocess while minimizing a total resource usage of all such assets. Theasset allocation server 2102 may receive and utilize data from sensors2112 at the data center and/or sensors 2120 in the data centerequipment. The asset allocation server 2102 may also receive data fromthe server agents 2116 and the various controllers 2114.

The asset allocation server 2102 may be performed using integratedcontrol across the system 2100 using the integrated optimizationprocesses of U.S. patent application Ser. No. 14/694,633, filed Apr. 23,2015, the entire disclosure of which is incorporated herein byreference. In some embodiments, the asset allocation server 2102 mayallocate loads across the system 2100 using the asset allocationprocesses of U.S. patent application Ser. No. 15/473,496, filed Mar. 29,2017, the entire disclosure of which is incorporated herein byreference. In some embodiments, the asset allocation server 2102executes process 1700, process 1800, process 1900, and/or process 2000.For example, the asset allocation process of U.S. patent applicationSer. No. 15/473,496, filed Mar. 29, 2017, the entire disclosure of whichis incorporated herein by reference, may be adapted to account for andcontrol operation of the data center equipment as described withreference to FIGS. 17-20 .

The asset allocation server 2102 may thereby generate control dispatchesfor the c central plant equipment 2104, airside system 2106, computerroom air conditioners 2108, rack cooling units 2110, and data centerequipment 1202, and transmit such dispatches to the controllers 2114 andthe server agents 2116. The system 2100 can thereby be operated in aunified manner for optimal performance.

In some embodiments, the asset allocation server 2102 is remote from thedata center. In other embodiments, the asset allocation server 2102 isincluded in the data center equipment 1202 and operates locally withinthe data center, such that the asset allocation server 2102 is among theservers for which temperature is managed by operation of the assetallocation server 2102.

Liquid Cooling for Data Center

Referring now to FIG. 22 , a schematic illustration of a system 2200that provides liquid cooling for a data center is shown, according to anexemplary embodiment. As described in detail above, the central plant 10is configured to generate chilled fluid (e.g., water) that can beprovide to a building, including to a data center. In the system 2200,the chilled fluid is used for liquid cooling of the data center. In theexample shown the central plant 10 provides chilled fluid to a datacenter via a pipe (or other fluid pathway) 2212. The fluid is alsoreturned to the central plant 10 via the pipe 2212.

In the example shown, the data center equipment 1202 includes multipleserver racks, shown as server rack A 2202, server rack B 2204, serverrack C 2206, server rack D 2208, and server rack E 2210. Each serverrack is in direct thermal contact with the pipe 2212. For example, eachserver rack may include a heat exchanger configured to provide a highefficiency of thermal energy transfer from the server rack to the pipe2212. In the illustration of FIG. 22 , server rack A 2202 providesthermal energy generated by devices at server rack A 2202, denoted asQ_(ServerA), to the chilled fluid from the central plant 10.Additionally, server rack B 2204 provides thermal energy generated bydevices at server rack B 2204, denoted as Q_(ServerB), to the chilledfluid from the central plant 10, server rack C 2206 provides thermalenergy generated by devices at server rack C 2206, denoted asQ_(ServerC), to the chilled fluid from the central plant 10, server rackD 2208 provides thermal energy generated by devices at server rack D2208, denoted as Q_(ServerD), to the chilled fluid from the centralplant 10, and server rack E 2210 provides thermal energy generated bydevices at server rack E 2210, denoted as Q_(ServerE), to the chilledfluid from the central plant 10. Although the racks 2202-2210 are shownas arranged in series along the pipe 2212, in other embodiments, thepipe 2212 is configure to interact with the racks 2202-2210 in parallel.

The fluid warms due to the thermal energy from the racks 2202-2210. Thereturn fluid is returned to the central plant 10, where it can bechilled again or otherwise allocated in the central plant 10.

Advantageously, the liquid cooling system 2200 is configured to benefitfrom all of the various features described above with reference to FIGS.1-11 . In particular, the chilled water load attributable to the liquidcooling system 2200 for the data center equipment 1202 can beincorporated into the central plant optimization systems and methods ofFIGS. 1-11 , and the central plant controller can be configured toallocate loads to cover the demands of the liquid cooling system 2200.The liquid cooling system 2200 may also be used with the systems andmethods of FIGS. 12-21 .

Controlling Heat and Task Distribution Across Data Center

Referring now to FIG. 23 , an illustration of heat and task distributionacross a data center is shown, according to an exemplary embodiment. Inparticular, FIG. 23 shows a heat map of a data center 2300. The heat mapshows multiple servers (data center equipment 1202), with theillustration indicating a relative temperature of each server. Forexample, server 2302, server 2304, server 2306, server 2308, and server2310 are shown as being hot, e.g., relative to a preset value abovewhich a server is classified as hot. As another example, server 2312 andserver 2314 are shown as being cool, e.g., relative to a preset valuebelow which a server is classified as being cool. Other servers areshown as medium temperature, e.g., at temperatures between the valuesthat define the hot and cold categories.

The data shown in the heat map of FIG. 23 may be collected by sensorspositioned in the servers and/or by multiple sensors or camerasdistributed around the data center 2300. The sensors can detect hotspots in the data center. Temperature differential across the serversmay be caused by differences in task allocation across the servers,hardware differences between the servers, proximity of servers toairflow, heat flow through walls or from building mass, the arrangementand operation of cooling equipment 2316, etc. The temperaturedifferential is considered across the area of the data center (e.g., thefloor) or across vertical levels (i.e., in two dimensions), or acrossvolumes in three-dimensions. Server locations and operations can beadjusted to reduce temperature gradations vertically and/or across thearea of the data center in some embodiments.

In some embodiments, different servers have different optimal operatingtemperatures. In such cases, the classification illustrated by the heatmap may be configured to adjust for such differences by independentlydefining the classification criteria for each server. Accordingly, eachserver may be classified as hot, medium, or cool based on the optimaloperating temperature or temperature range of the corresponding server.A server-by-server thermal control may be implemented based on aparticular temperature setpoint for each particular server.

As illustrated in FIG. 23 , the cooling equipment 2316 (e.g., computerroom air conditioners, rack cooling units, server fans, airside system,liquid cooling system) that serves the data center 2200 may beconfigured to provide targeted cooling to the servers classified asbeing hot. In the example shown, the cooling equipment 2316 providestargeted cooling to hot servers 2302-2306. By targeting the cooling,energy is saved from being used to needlessly affect the temperature ofother servers which are already maintained at an acceptable temperaturelevel. Additionally, cooling can be directed as need to control todifferent setpoints for different servers.

Also as illustrated in FIG. 23 , in some cases the data center 2300 iscontrolled to shift tasks from hot servers to cool- ormedium-temperature servers. In the example shown, tasks originallyscheduled to be executed by server 2308 are reassigned to server 2314,while tasks originally scheduled to be executed by server 2310 arereassigned to server 2312. The heat generation associated with executionof these tasks is thereby also shifted from the hot servers 2308-2310 tocool servers 2312-2314, potentially achieving a similar heat-reductionas that which would result from targeted cooling. Heat may thereby bespread more evenly around the data center 2300, which may reduce therisk of overheating of any individual server and reduce the need fortargeted cooling. This may increase the reliability and efficiency ofall servers in the data room as well as reduce the load on the coolingequipment 2316. Additionally, task shifting may facilitate controllingtemperature to different setpoints for different servers.

In various embodiments, coordination of targeted cooling from thecooling equipment 2316 and task shifting between servers is performed,for example by integrated controller 1500 of FIG. 15 or asset allocationserver 2102 of FIG. 21 . One goal for balancing the use of targetedcooling and task shifting may be an attempted to establish anequilibrium across the data room. In some cases, a temperature gradientacross the data room is minimized by task shifting and control of thecooling equipment 2316. In some embodiments, process 2000 is adapted toprovide for control on a per-server basis that optimizes overallresource usage.

In some embodiments, the data center includes multiple temperaturesensors disposed at multiple spots in the data center. The sensors candetect hot spots within the data center (e.g., due to operations of oneor more servers). In some embodiments, the hot spots are detected usingone or more thermal cameras in the data room. The operations performedby the one or more servers in the hot spots are shifted by the loadcontrol agent or other controller to other servers outside of the hotspots in some embodiments. In some embodiments, servers in the coolerand/or coolest areas of the rooms are chosen for the operations shedform the servers in the hot spots. In some embodiments, the servers aredeployed in movable racks on automated carts and the carts are movedoutside of the hot spots to cooler area in the data room in response toa detection of a hot spot. In some embodiments, the load control agentadjusts the loads on the server or the placement of the servers toachieve a thermal equilibrium throughout the data center. In someembodiments, the cooling equipment may directly provide cool air orliquid to the hot spot location in response to hot spot detection.

Various adaptations of the systems and methods described herein tofacilitate server-by-server monitoring and control are within the scopeof the present disclosure.

Configuration of Exemplary Embodiments

The construction and arrangement of the systems and methods as shown inthe various exemplary embodiments are illustrative only. Although only afew embodiments have been described in detail in this disclosure, manymodifications are possible (e.g., variations in sizes, dimensions,structures, shapes and proportions of the various elements, values ofparameters, mounting arrangements, use of materials, orientations,etc.). For example, the position of elements may be reversed orotherwise varied and the nature or number of discrete elements orpositions may be altered or varied. Accordingly, all such modificationsare intended to be included within the scope of the present disclosure.The order or sequence of any process or method steps may be varied orre-sequenced according to alternative embodiments. Other substitutions,modifications, changes, and omissions may be made in the design,operating conditions and arrangement of the exemplary embodimentswithout departing from the scope of the present disclosure.

The present disclosure contemplates methods, systems and programproducts on memory or other machine-readable media for accomplishingvarious operations. The embodiments of the present disclosure may beimplemented using existing computer processors, or by a special purposecomputer processor for an appropriate system, incorporated for this oranother purpose, or by a hardwired system. Embodiments within the scopeof the present disclosure include program products or memory comprisingmachine-readable media for carrying or having machine-executableinstructions or data structures stored thereon. Such machine-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer or other machine with a processor.By way of example, such machine-readable media can comprise RAM, ROM,EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to carry or store desired program code in the form ofmachine-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer or othermachine with a processor. Combinations of the above are also includedwithin the scope of machine-readable media. Machine-executableinstructions include, for example, instructions and data which cause ageneral purpose computer, special purpose computer, or special purposeprocessing machines to perform a certain function or group of functions.

As used herein, the term “circuit” may include hardware structured toexecute the functions described herein. In some embodiments, eachrespective “circuit” may include machine-readable media for configuringthe hardware to execute the functions described herein. The circuit maybe embodied as one or more circuitry components including, but notlimited to, processing circuitry, network interfaces, peripheraldevices, input devices, output devices, sensors, etc. In someembodiments, a circuit may take the form of one or more analog circuits,electronic circuits (e.g., integrated circuits (IC), discrete circuits,system on a chip (SOCs) circuits, etc.), telecommunication circuits,hybrid circuits, and any other type of “circuit.” In this regard, the“circuit” may include any type of component for accomplishing orfacilitating achievement of the operations described herein. Forexample, a circuit as described herein may include one or moretransistors, logic gates (e.g., NAND, AND, NOR, OR, XOR, NOT, XNOR,etc.), resistors, multiplexers, registers, capacitors, inductors,diodes, wiring, and so on).

The “circuit” may also include one or more processors communicablycoupled to one or more memory or memory devices. In this regard, the oneor more processors may execute instructions stored in the memory or mayexecute instructions otherwise accessible to the one or more processors.In some embodiments, the one or more processors may be embodied invarious ways. The one or more processors may be constructed in a mannersufficient to perform at least the operations described herein. In someembodiments, the one or more processors may be shared by multiplecircuits (e.g., circuit A and circuit B may comprise or otherwise sharethe same processor which, in some example embodiments, may executeinstructions stored, or otherwise accessed, via different areas ofmemory). Alternatively or additionally, the one or more processors maybe structured to perform or otherwise execute certain operationsindependent of one or more co-processors. In other example embodiments,two or more processors may be coupled via a bus to enable independent,parallel, pipelined, or multi-threaded instruction execution. Eachprocessor may be implemented as one or more general-purpose processors,application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital signal processors (DSPs), or other suitableelectronic data processing components structured to execute instructionsprovided by memory. The one or more processors may take the form of asingle core processor, multi-core processor (e.g., a dual coreprocessor, triple core processor, quad core processor, etc.),microprocessor, etc. In some embodiments, the one or more processors maybe external to the apparatus, for example the one or more processors maybe a remote processor (e.g., a cloud based processor). Alternatively oradditionally, the one or more processors may be internal and/or local tothe apparatus. In this regard, a given circuit or components thereof maybe disposed locally (e.g., as part of a local server, a local computingsystem, etc.) or remotely (e.g., as part of a remote server such as acloud based server). To that end, a “circuit” as described herein mayinclude components that are distributed across one or more locations.The present disclosure contemplates methods, systems and programproducts on any machine-readable media for accomplishing variousoperations. The embodiments of the present disclosure can be implementedusing existing computer processors, or by a special purpose computerprocessor for an appropriate system, incorporated for this or anotherpurpose, or by a hardwired system. Embodiments within the scope of thepresent disclosure include program products comprising machine-readablemedia for carrying or having machine-executable instructions or datastructures stored thereon. Such machine-readable media can be anyavailable media that can be accessed by a general purpose or specialpurpose computer or other machine with a processor. By way of example,such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROMor other optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to carry or storedesired program code in the form of machine-executable instructions ordata structures and which can be accessed by a general purpose orspecial purpose computer or other machine with a processor. Combinationsof the above are also included within the scope of machine-readablemedia. Machine-executable instructions include, for example,instructions and data which cause a general purpose computer, specialpurpose computer, or special purpose processing machines to perform acertain function or group of functions.

Although the figures may show a specific order of method steps, theorder of the steps may differ from what is depicted. Also two or moresteps may be performed concurrently or with partial concurrence. Suchvariation will depend on the software and hardware systems chosen and ondesigner choice. All such variations are within the scope of thedisclosure. Likewise, software implementations could be accomplishedwith standard programming techniques with rule based logic and otherlogic to accomplish the various connection steps, processing steps,comparison steps and decision steps.

What is claimed is:
 1. A method for operating a heating, ventilation, orair conditioning (HVAC) system in a data center, comprising: removingheat from air in the data center utilizing the HVAC system, the HVACsystem comprising at least one of a central plant, an airside system, awaterside system, rack cooling equipment, a computer-room airconditioner, a rooftop unit, a floor cooling system, or a liquid coolingsystem; detecting a hot spot within the data center, the hot spot beingcaused by operations of one or more servers in the data center; anddirectly providing cooling to the hot spot in response to detecting thehot spot.
 2. The method of claim 1, further comprising: using spacetemperature data, server temperature data, and a model of performancefor the HVAC system and the data center to predict changes to the HVACsystem and the data center, the changes predicted to conserve energywhile complying with temperature constraints for the data center andmeeting a processing demand on the data center; and electronicallycontrolling the HVAC system and the data center in accordance with thechanges for both the HVAC system and the data center.
 3. The method ofclaim 2, wherein the model of performance of the HVAC system and thedata center models an operational efficiency of servers of the datacenter as a function of at least one of the space temperature data orthe server temperature data.
 4. The method of claim 1, wherein the hotspot is detected using one or more thermal cameras in the data center.5. The method of claim 1, wherein more than one hot spot is detected andthe cooling is provided to the more than one hot spot and the methoduses artificial intelligence.
 6. The method of claim 1, furthercomprising: shifting operations performed by the one or more servers inthe hot spot to other servers outside of the hot spot.
 7. The method ofclaim 6, wherein the other servers are in a cooler location of the datacenter.
 8. The method of claim 1, further comprising: providing loadinformation for the data center to a user interface.
 9. A systemcomprising: servers located at a data center; equipment configured tocool the servers, the equipment comprising at least one of a centralplant, an airside system, a waterside system, rack cooling equipment, acomputer-room air conditioner, a rooftop unit, a floor cooling system,or a liquid cooling system; a controller configured to control theequipment based on estimates of an amount of thermal energy output byeach of the servers and energy cost.
 10. The system of claim 9, whereinthe controller is configured to apply the estimates of the amount ofthermal energy output by the servers in a feedforward control approachto generate setpoints for the equipment or is configured to useartificial intelligence.
 11. The system of claim 10, wherein theestimates of the amount of thermal energy output by the servers comprisepredictions of the amount of thermal energy output by the servers for aplurality of time steps; and wherein the controller is configured togenerate the setpoints for the equipment for the plurality of time stepsbased on the predictions.
 12. The system of claim 11, wherein thecontroller is configured to generate the setpoints by performing anoptimization process that minimizes an overall resource usage of theservers and the equipment over a time horizon.
 13. The system of claim9, wherein the controller is configured to reschedule operations of theservers based on a time-variant cost of operating the equipment.
 14. Thesystem of claim 9, wherein the servers comprise fans and the controlleris configured to control the fans.
 15. The system of claim 14, whereinthe controller is configured to coordinate control of the watersidesystem, the airside system, the rack-cooling system, and the fans.
 16. Amethod comprising: controlling operations of servers at a data center;controlling building equipment to cool the servers; determining ahigh-cost period for operating the building equipment and a low-costperiod for operating the building equipment; identifying atime-insensitive task scheduled to occur during the high-cost period;and causing the time-insensitive task to be executed during the low-costperiod.
 17. The method of claim 16, comprising: predicting ahigh-activity period for the servers; and operating the buildingequipment to reduce a temperature at the servers before thehigh-activity period.
 18. The method of claim 16, comprising shiftingthe time-insensitive task from a first server to a second server basedon a temperature differential between the first server and the secondserver.
 19. The method of claim 16, wherein determining the high-costperiod comprises predicting weather-based loads on the buildingequipment.
 20. The method of claim 16, wherein determining the high-costperiod and the low-cost period comprises determining utility rates overa time horizon and wherein the building equipment comprises at least oneof a central plant, an airside system, a waterside system, rack coolingequipment, a computer-room air conditioner, a rooftop unit, a floorcooling system, or a liquid cooling system.